Project

Welcome to an interactive exploration of themes and topics in Charlotte Brontë's Jane Eyre. This project leverages topic modeling techniques to uncover thematic structures within the novel. By analyzing how different chapters contribute to identified topics, we gain a deeper understanding of the novel's underlying themes and their evolution throughout the story.

The purpose of this explorable explanation is to guide users into the world of sentiment and topic analysis. Using a fine-tuned Latent Dirichlet Allocation (LDA) model, we can infer themes in a novel such as Jane Eyre. The results produced by the model are significant, especially for those familiar with the novel, as they help reveal potential themes and offer insights into the text's structure and content. For those unfamiliar with the novel, this exploration provides a learning opportunity about its themes and underlying messages.

Before diving into the analysis, the text needs to be preprocessed. This involves tokenizing the chapters into words, creating bigrams and trigrams, and preparing the data for the LDA model. Tokenization breaks the text into individual words. Bigrams and trigrams are phrases of two and three words respectively, which help in capturing common phrases and improving the topic modeling results.

# Tokenize chapters
tokenized_chapters = [preprocess(chapter) for chapter in chapters]

# Build bigrams and trigrams
bigram = Phrases(tokenized_chapters, min_count=5, threshold=100)
trigram = Phrases(bigram[tokenized_chapters], threshold=100)

tokenized_chapters = [trigram[bigram[chapter]] for chapter in tokenized_chapters]

Latent Dirichlet Allocation (LDA) is a topic modeling technique that identifies groups of words (topics) that frequently occur together. I am using LDA to discover themes in the text of Jane Eyre.

The Intertopic Distance Map shows the relationships between the topics. Each circle represents a topic, and the size of the circle indicates its prevalence. The proximity of the circles reflects the similarity between the topics. The x and y axes represent the principal component values that capture the distances between topics.

The Top-30 Most Relevant Terms for Topic visualization shows the top 30 terms for each topic. The x-axis represents the relevance score, and the y-axis lists the terms. Adjusting the relevance metric slider can emphasize terms that are more unique to the topic or those that are more frequent in the overall text.

# Prepare data for pyLDAvis
vis = genvis.prepare(optimal_model, corpus, dictionary)

# Save the visualization as an HTML file
pyLDAvis.save_html(vis, 'lda_visualization.html')

Explanation: This code prepares the data for visualization using pyLDAvis and saves it as an HTML file. The visualizations help in understanding the inter-topic distances and the relevance of terms within topics.

User Instructions:

• Hover over the circles in the Intertopic Distance Map to see the topic number and its prevalence. Click on a circle to focus on that topic and see detailed information.
• Use the relevance metric slider in the Top-30 Terms visualization to adjust the weight of term relevance. Hover over terms to see their contribution to the topic, and click on terms to see their usage in the context of the novel.

To evaluate the quality of the topics generated by the LDA model, we calculate the coherence score. The coherence score measures the degree of semantic similarity between high scoring words in a topic. Higher coherence scores indicate more interpretable and meaningful topics.

In this project, I used the Coherence Model from Gensim to compute the coherence score for our LDA model. The coherence score helps us assess the quality of the topics and decide on the optimal number of topics for our analysis.

# Calculate coherence score for LDA model
from gensim.models import CoherenceModel

coherence_model_lda = CoherenceModel(model=optimal_model, texts=tokenized_chapters, dictionary=dictionary, coherence='c_v')
coherence_lda = coherence_model_lda.get_coherence()

print('\nCoherence Score: ', coherence_lda)

Explanation: This code calculates the coherence score for the LDA model. The score helps in evaluating the semantic similarity of the topics.

To visualize how coherence scores change with different numbers of topics, we can plot the coherence score for a range of topic numbers. This helps in identifying the optimal number of topics that produce the highest coherence score.

# Compute coherence scores for different numbers of topics
coherence_scores = []
for num_topics in range(2, 40, 2):
    model = LdaModel(corpus=corpus, id2word=dictionary, num_topics=num_topics, random_state=100, update_every=1, chunksize=100, passes=10, alpha='auto', per_word_topics=True)
    coherence_model = CoherenceModel(model=model, texts=tokenized_chapters, dictionary=dictionary, coherence='c_v')
    coherence_scores.append((num_topics, coherence_model.get_coherence()))

# Prepare data for Plotly visualization
x = list(range(2, 40, 2))
data = {
    "Number of Topics": x,
    "Coherence Score": [score for num_topics, score in coherence_scores]
}

df = pd.DataFrame(data)

# Create an interactive Plotly graph
fig = px.line(df, x="Number of Topics", y="Coherence Score", markers=True,
              title="Coherence Scores for Different Numbers of Topics",
              labels={"Number of Topics": "Number of Topics", "Coherence Score": "Coherence Score"},
              template="plotly_white")

# Customize the layout to make it visually appealing
fig.update_traces(line={color: "#A979B9", width: 4}, marker={size: 10, color: "#A979B9"})
fig.update_layout(
    title_font={size: 20, family: "Arial", color: "#4A4A4A"},
    xaxis={title_font: {size: 16, family: "Arial", color: "#4A4A4A"},
               tickfont: {size: 12, family: "Arial", color: "#4A4A4A"}},
    yaxis={title_font: {size: 16, family: "Arial", color: "#4A4A4A"},
               tickfont: {size: 12, family: "Arial", color: "#4A4A4A"}},
    hovermode: "x unified",
    hoverlabel: {font_size: 14, font_family: "Arial"},
    margin: {l: 50, r: 50, t: 80, b: 50},
    plot_bgcolor: "#FFFFFF",
    paper_bgcolor: "#FFFFFF",
    showlegend: False
)

# Save the plot as an HTML file
fig.write_html("coherence_scores.html")

# Show the plot
fig.show()

Explanation: This code computes the coherence scores for different numbers of topics and creates an interactive Plotly graph to visualize the scores. The plot helps in identifying the number of topics that maximizes the coherence score.

User Instructions:

• Hover over the line plot to see the exact coherence score for each number of topics.
• Use the interactive features to zoom in and out for a closer look at specific ranges.

The following sections provide a detailed analysis of the topics and their associated themes, the top positive and negative topics, and the chapter contributions to each topic.

Inferred Themes

Based on the analysis, the following themes were inferred for each topic:

• Topic 0: Death and Relationships
• Topic 1: Struggle and Survival
• Topic 2: Education and Society
• Topic 3: Illness and Reflection
• Topic 4: Social Interactions and Beauty
• Topic 5: Family and Wealth
• Topic 6: Nature and Romance
• Topic 7: Loss and Remembrance
• Topic 8: Childhood and Sadness
• Topic 9: Education and Freedom
• Topic 10: Social Hierarchy and Beauty
• Topic 11: Marriage and Human Nature
• Topic 12: Places and Freedom
• Topic 13: Emotions and Experience
• Topic 14: Friendship and Beauty
• Topic 15: Home and Calmness
• Topic 16: Daily Life and Education
• Topic 17: Home and Relationships

Top Positive and Negative Topics

Based on the sentiment analysis, here are the top positive and negative topics:

Top Positive Topics

• Topic 4: Adèle, Mrs. Fairfax, beauty, position
• Topic 6: Thornfield, moon, sir, rain, road
• Topic 10: Ladies, servants, gentleman, mrs. fairfax, adèle
• Topic 15: Fairfax, thornfield, road, mrs. fairfax, calm
• Topic 12: Lowood, thornfield, ladies, french, liberty

Top Negative Topics

• Topic 1: Die, aid, rain, servant, human
• Topic 11: Wife, human, calm, heaven, suddenly
• Topic 17: Mrs. Fairfax, thornfield, front, adèle, chamber
• Topic 8: Children, french, broken, sad, frame
• Topic 0: Sisters, dead, mere, blood, demanded

Sentiment analysis assesses the emotional tone of the text. I applied sentiment analysis to each topic to understand the emotional tone associated with each theme. The Sentiment Distribution Bar Chart visualizes the sentiment scores of the topics, showing the count of positive and negative sentiments associated with each topic.

The x-axis represents the topics, while the y-axis represents the count of chapters associated with each sentiment. The bars are color-coded: lightblue for positive and darkgray for negative sentiments.

The neutral sentiment count is zero due to the chosen sentiment threshold values. By setting the thresholds for positive sentiment to scores above 0.5 and negative sentiment to scores below -0.5, I aimed to clearly differentiate between strongly positive and strongly negative sentiments. This approach helps in highlighting the extremes and reduces the ambiguity that often accompanies neutral sentiment. This choice was intentional to focus on the distinct emotional tones and their impacts on the themes within the novel.

# Calculate average sentiment scores for each topic
topic_sentiments = np.zeros(num_topics)
topic_token_counts = np.zeros(num_topics)
topic_chapter_counts = np.zeros(num_topics)

for i, (chapter, label) in enumerate(chapters_with_labels):
    topic_distribution = optimal_model.get_document_topics(corpus[i], minimum_probability=0)
    for topic_id, probability in topic_distribution:
        sentiment = sia.polarity_scores(' '.join(chapter))[compound]
        topic_sentiments[topic_id] += sentiment * probability
        topic_token_counts[topic_id] += len(chapter) * probability
        topic_chapter_counts[topic_id] += probability

# Normalize the sentiment scores
average_topic_sentiments = topic_sentiments / np.where(topic_chapter_counts == 0, 1, topic_chapter_counts)

Explanation: This code calculates the average sentiment scores for each topic using the Vader Sentiment Analysis tool. The sentiment scores are then normalized based on the topic distribution.

# Create a DataFrame for sentiment distribution
sentiment_counts = []
for topic in range(num_topics):
    positive_count = len([sent for sent in chapter_sentiments if sent > 0.5])
    neutral_count = len([sent for sent in chapter_sentiments if -0.5 <= sent <= 0.5])
    negative_count = len([sent for sent in chapter_sentiments if sent < -0.5])
    sentiment_counts.append((topic, 'Positive', positive_count))
    sentiment_counts.append((topic, 'Neutral', neutral_count))
    sentiment_counts.append((topic, 'Negative', negative_count))

df_sentiments = pd.DataFrame(sentiment_counts, columns=['Topic', 'Sentiment', 'Count'])

# Create the bar chart with Plotly
fig = px.bar(
    df_sentiments,
    x='Topic',
    y='Count',
    color='Sentiment',
    barmode='group',
    title='Sentiment Distribution Across Topics in Jane Eyre',
    color_discrete_map={'Positive': 'blue', 'Neutral': 'grey', 'Negative': 'red'}
)

# Customize the layout
fig.update_layout(
    xaxis_title='Topic',
    yaxis_title='Count of Sentiments',
    legend_title='Sentiments'
)

# Show the plot
fig.show()

# Save the plot as an HTML file
fig.write_html("sentiment_distribution.html")

Explanation: This code creates a grouped bar chart to visualize the sentiment distribution across topics. It shows the count of positive and negative sentiments for each topic, making it easier to understand the emotional tone associated with each theme.

User Instructions: Use the interactive bar chart to explore the sentiment distribution across topics. Hover over the bars to see the count of sentiments for each topic. The color-coded bars help in distinguishing between positive and negative sentiments.

The Topic Evolution Line Chart provides a dynamic visualization of how the prevalence of each topic changes over the chapters of Jane Eyre. By observing these trends, we can gain insights into the development and transformation of themes throughout the narrative.

The x-axis represents the chapter numbers, allowing you to track the progression of the story, while the y-axis represents the probability of each topic being prominent in the corresponding chapter. This setup enables us to see how certain themes emerge, peak, and fade as the story unfolds.

The chart includes a dropdown menu for selecting specific topics. By choosing a topic from the dropdown, you can isolate its trend across the chapters, making it easier to focus on individual thematic developments.

In particular, the topics "Loss and Remembrance" and "Emotions and Experience" show a straight horizontal line at a small non-zero value. This indicates that these topics have a consistent, low-level presence throughout the novel. They act as background themes, providing subtle but constant emotional and experiential elements to the narrative without becoming the main focus at any point.

# Prepare data for topic distribution over chapters
topic_distributions = []

for i, (chapter, label) in enumerate(chapters_with_labels):
    topic_distribution = optimal_model.get_document_topics(corpus[i], minimum_probability=0)
    chapter_distribution = {'Chapter': i+1}
    for topic_id, probability in topic_distribution:
        chapter_distribution['Topic ' + topic_names[topic_id]] = probability
    topic_distributions.append(chapter_distribution)

df_topic_distributions = pd.DataFrame(topic_distributions).fillna(0)

# Melt the DataFrame for better plotting with Plotly
df_melted = df_topic_distributions.melt(id_vars=['Chapter'], var_name='Topic', value_name='Probability')

# Create the dropdown menu options
dropdown_buttons = [
    {label: topic, method: 'update', args: [{visible: [t == topic for t in df_melted['Topic'] .unique()]}, {title: 'Topic Evolution for ' + topic]}]} for topic in topic_names
]

# Add a default option
dropdown_buttons.insert(0, {label: 'Choose a topic :)', method: 'update', args: [{visible: [False] * len(df_melted['Topic'].unique())}, {title: 'Topic Evolution Over Chapters in Jane Eyre'}]})

# Create the interactive line chart with dropdown
fig = go.Figure()

for topic in df_melted[Topic].unique():
    fig.add_trace(go.Scatter(
        x=df_melted[Chapter][df_melted[Topic] == topic],
        y=df_melted[Probability][df_melted[Topic] == topic],
        mode=lines,
        name=topic,
        line={color: #A979B9},  # Set line color to lilac
        visible=False  # Initially set all traces to be invisible
    ))

# Add the dropdown menu
fig.update_layout(
    updatemenus=[
        {
            buttons: dropdown_buttons,
            direction: down,
            showactive: True,
            x: 1.01,
            xanchor: left,
            y: 1.02,
            yanchor: top,
            bgcolor: #A979B9,  # Darker lilac background
            bordercolor: #985FAB,  # Darker shade of lilac border
            font: {color: white, size: 14},  # Font settings
            borderwidth: 2,
            pad: {r: 0, t: 10},
            active: 0  # Ensure the default option is highlighted
        }
    ],
    xaxis_title='Chapter In The Novel',
    yaxis_title='Topic Probability',
    xaxis={range=[0.01, df_melted[Chapter].max()],
        tickmode=linear,
        dtick=5,
        showgrid=True,
        zeroline=True,
        zerolinewidth=2,
    },
    yaxis={range=[0, df_melted[Probability].max()],
        tickmode=linear,
        dtick=0.1,
        showgrid=True,
        zeroline=True,
    },
    title={text: Topic Evolution Over Chapters in Jane Eyre,
        x=0.5,
        xanchor=center,
        font={size: 20, color: black, family: Arial}
    },
    margin={l: 50, r: 200, t: 100, b: 50},  # Adjust margins as needed
    legend={x: 1.2,
        y: 0.5,
        title={text: Topics}
    }
)

# Show the plot
fig.show()

# Save the interactive plot as an HTML file
fig.write_html("topic_evolution.html")

Explanation: This code prepares the data for visualizing the topic distribution over chapters. The data is then melted for better plotting with Plotly.

User Instructions:

• Use the dropdown menu located to the right of the chart to select a specific topic. The chart will update to show the trend of the chosen topic across the chapters.
• Hover over the lines in the chart to see detailed information about the topic probabilities for each chapter.
• Pay attention to the x-axis (Chapter In The Novel) and y-axis (Topic Probability) to understand how the prominence of each topic evolves throughout the story.

This visualization aims to convey how different themes are distributed and how they change as the narrative progresses. By isolating individual topics, you can gain deeper insights into the thematic structure of Jane Eyre and how specific themes contribute to the overall story.

For each topic, I provide an overview of the significant chapters contributing to each topic, along with their corresponding content from the novel. This section helps us evaluate how accurately the model has captured the themes within the text.

The highlighted text passages provide an interactive way to explore how specific words contribute to the identified topics. Use the dropdown to select a topic and see the corresponding text passages highlighted based on the topic's word probabilities. This allows for a more granular examination of how themes are distributed throughout the text.

User Instructions: Select a topic from the dropdown menu to see the text passages where the words contributing to that topic are highlighted. This interactive feature helps you understand how the identified themes are distributed throughout the novel's text.

This project demonstrates the power of topic modeling in uncovering the thematic structure of a literary work. By applying LDA to Jane Eyre, I have identified key themes and tracked their evolution throughout the novel. The interactive visualizations provide a dynamic way to explore these themes and gain deeper insights into Charlotte Brontë's masterpiece.

The results show a strong alignment between the generated topics and the novel's content, validating the effectiveness of our approach. This interactive exploration not only enhances our understanding of the novel but also showcases the potential of data visualization and natural language processing in literary analysis.