Week 8. Feb. 28: Auto-encoders, Network & Table Learning - Possibilities

[avioberoi]

Post a link for a "possibility" reading of your own on the topic of Auto-encoders, Network & Table Learning [Week 8], accompanied by a 300-400 word reflection that:

1. Briefly summarizes the article (e.g., as we do with the first “possibility” reading each week in the syllabus)
2. Suggests how its method could be used to extend social science analysis
3. Describes what social data you would use to pilot such a use with enough detail that someone could move forward with implementation

[psymichaelzhu]

(Kim et al., 2024)

1. Summary of the Article
   This study explores how disruptions in personal routines, such as travel and COVID-19 lockdowns, influence user musical exploration. Using a dataset with 63.3 million listening sessions of 496 million songs, the researchers find that when individuals experience disruptions in their daily routines, they tend to explore new music. Travelers adapt their musical preferences toward their destination’s regional music, while lockdowns push listeners toward greater musical diversity, but not necessarily toward global mainstream music. The findings support Swidler’s theory of “unsettled times”, suggesting that periods of instability lead to cultural innovation and long-term shifts in preferences.

2. Extending the Method to Social Science Analysis
   Besides user taste, the Song2Vec (S2V) model could also be applied to study artists' innovation.
   Creative evolution of artists: By tracking how artists' song embeddings shift over personal or overall history, we could model the trajectory of artistic innovation, identifying whether disruptions in their personal or professional lives (like the emergence of streaming platform) correlate with stylistic changes.
   Unlike feature-based approaches that rely on predefined features like audio characteristics (e.g., tempo, or loudness), this relation-based method captures contextual similarity between tracks without requiring explicit feature definitions. This flexibility allows for a more data-driven understanding of musical connections, potentially revealing creative patterns that traditional feature-based methods might overlook.

3. Social Data for Pilot Study
   To implement this approach, we will combine historical music data and artist metadata to examine how musicians’ creative trajectories evolve in response to personal or external disruptions.

Data Sources:
a. Spotify’s 1921-2020 dataset (Kaggle): Contains 600,000+ tracks with metadata like artist and release date, offering a broad historical perspective on music evolution.
b. Spotify Playlists dataset (Kaggle): Includes 12.9 million track recordings across 162,000 unique playlists, providing contextual relationships between songs.
c. Historical & Biographical Data (maybe from Wikipedia):

• Artist biographical information to track personal events such as relocations or career disruptions.
• Historical disruptions (e.g., World War II, the rise of streaming platforms) to analyze their influence on global music trends.

Analysis Plan:
a. Train a Song2Vec (S2V) model using the playlist dataset to embed songs in a continuous space based on contextual similarity.
b. Construct artist trajectories by computing the centroid of their song embeddings over time.
c. Identify significant shifts in trajectories and correlate them with historical or personal events, such as technological shifts, e.g., streaming platforms.
d. Quantitatively test the impact of disruption events using methods such as interrupt time series analysis.
This approach enables a data-driven exploration of musical evolution, revealing how external disruptions shape artistic creativity over time.

[zhian21]

Perozzi, B., Al-Rfou, R., & Skiena, S. (2014, August). Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 701-710).

DeepWalk introduces a novel method for learning latent representations of network nodes by leveraging truncated random walks and applying unsupervised learning techniques inspired by language modeling. By treating random walks as sequences analogous to sentences, DeepWalk embeds graph structures into a continuous vector space, enabling improved performance on multi-label classification tasks, particularly in sparse-label environments. The approach significantly outperforms traditional network classification baselines such as spectral clustering and modularity-based methods, demonstrating up to 10% higher F1 scores while requiring substantially less labeled data. Additionally, DeepWalk is scalable and parallelizable, making it applicable to large social networks.

This method could extend social science analysis by providing a scalable and effective way to model and analyze social influence, community structures, and opinion dynamics. By embedding individuals in a continuous space based on their network interactions, DeepWalk enables researchers to capture latent group memberships, detect ideological clusters, and predict behavioral trends without requiring explicit label supervision. Moreover, its adaptability to evolving networks allows for longitudinal studies of social change, polarization, and diffusion of beliefs.

To pilot this approach in social science, I would use the World Values Survey (WVS) Wave 7 dataset, which captures social attitudes, political beliefs, and demographic characteristics of individuals across diverse countries. By constructing a network graph where nodes represent respondents and edges capture ideological similarity (based on cosine similarity of responses or self-reported social ties), DeepWalk could generate embeddings reflecting ideological proximity. These embeddings could then be used to predict opinion shifts, polarization, and demographic correlates of belief structures, providing a richer understanding of how social attitudes cluster and evolve over time.

[yangyuwang]

Djeddou, Messaoud, Jehad Al Dallal, Aouatef Hellal, Ibrahim A. Hameed, and Xingang Zhao. 2024. “Hybrid Deep Convolutional Neural Networks Combined with Autoencoders and Augmented Data to Predict the Look-Up Table 2006.” Pp. 1–5 in 2024 International Conference on Decision Aid Sciences and Applications (DASA).

1. Summary of the Article
This study introduces a hybrid predictive model that integrates auto-encoders (AEs) with deep convolutional neural networks (DCNNs) for forecasting complex tabular data. The authors employ data augmentation techniques to enhance the robustness of predictions, particularly in the context of a computationally intensive lookup table (LUT) from 2006. The model leverages auto-encoders to reduce dimensionality and extract significant features from high-dimensional datasets, which are then processed by DCNNs to refine prediction accuracy. The paper demonstrates that this hybrid model improves performance compared to standalone deep learning methods, especially in cases where structured tabular data needs to be learned effectively.

2. Extending Social Science Analysis
The methodology from this paper could be adapted for social science research that involves structured tabular datasets, such as survey responses, economic indicators, or social network analytics. Auto-encoders offer a powerful means of feature extraction from high-dimensional data, allowing researchers to uncover latent patterns and relationships in social datasets. Additionally, integrating convolutional networks into tabular data analysis could enhance predictive modeling in fields like political forecasting, social mobility studies, and digital sociology.

3. Social Data for Pilot Implementation
To implement this method in a social science context, I propose using the World Values Survey (WVS) dataset, which contains structured responses on cultural, social, and political attitudes across different countries and time periods.
a. Auto-Encoder Processing: Compress the high-dimensional WVS data, identifying key latent variables that represent core social attitudes.
b. DCNN Integration: Use convolutional layers to predict societal trends, such as shifts in democratic support or trust in government institutions.
c. Data Augmentation: Apply synthetic data generation techniques to enhance model robustness, ensuring better generalization across different survey waves.

[haewonh99]

https://arxiv.org/abs/2407.03568

1. Summary
   When LLM Meets Hypergraph: A Sociological Analysis on Personality via Online Social Networks discusses mapping personality out of fractured, low-quality dataset on the internet. For this task, the researchers utilize LLMs and hypergraphs-graphs that connect multiple nodes, instead of traditional networks where edges only connect two nodes at a time. Specifically, the researchers designed an LLM prompt to consolidate information that the user had left in an online space and create more sophisticated profiles. Then, they mapped these descriptions in vector spaces to create a hypergraph out of users as nodes and social environments as hyper edges. The model was effective in predicting personality traits, MBTIs of the users.
   https://arxiv.org/abs/2407.03568

2. Application to Research
   While personality is an interesting variable to study indeed, as we are creating complicated character data from people in the network and are creating more complex profiles per each person, maybe we can identify more complex questions like political opinion on an issue or whether a consumer would buy a product.

To illustrate the latter of the two ideas - I think social media data like in this paper would suffice, although the tag would have to be the list of items purchased instead of MBTI or other personality traits measures. The researcher could go on and create a detailed profile of each person with LLM, create embeddings, builda hypergraph like this paper, train the model on whether the item of interest was in the purchased item of the user, and test the model efficiency on the test data.

[DotIN13]

Detecting coalitions by optimally partitioning signed networks of political collaboration Aref and Neal (2020) propose a mathematical optimization model for partitioning signed political collaboration networks into cohesive coalitions. Their approach minimizes intra-group negative and inter-group positive edges, solving the NP-hard problem of optimal partitioning using bounds and linear programming techniques. They apply this model to the U.S. Congress from 1979 to 2016, demonstrating that increasing polarization, as reflected in partisan coalitions, has been a protective factor in maintaining legislative effectiveness. Their findings challenge the assumption that polarization necessarily reduces legislative productivity, instead showing that ideologically homogeneous coalitions can streamline bill passage in the House of Representatives.

This method provides a powerful tool for analyzing political and social networks beyond legislative contexts. It could be applied to other domains where polarization and coalition formation play a role, such as corporate governance (e.g., boardroom alliances), international relations (e.g., diplomatic blocs in the United Nations), or social media discourse (e.g., ideological echo chambers). By identifying cohesive opposition groups, we can assess the impact of factionalization on decision-making efficiency, cooperation, and conflict resolution. Additionally, this approach could be integrated into research on organizational behavior, identifying workplace factions that impact collaboration or conflict, or in studies of activist movements, determining how ideological divides influence the effectiveness of policy advocacy.

In particular, this research highlights the potential of integrating the notion of signed network into the analysis of elite institutions. This conceptual construct can be easily adapted to the foreign policy decision-making scenario, where groups of policymakers constitute a signed elite network where nodes represent individuals and edge weights indicate homogeneity in policy positions. Signed graph embedding might offer a viable measure to the potential influential power of an individual, for instance, a respectful advisor who enjoys many positive in-degrees might be particularly persuasive in the decision-making process. Additionally, the methodology proposed in this paper can also be possibly used to identify coalitions in a foreign policy administration. The presence of a coalition at a policy meeting might find it easier to steer the policy decision in their own favor.

[chychoy]

Can Graphical Neural Networks Learn Language with Extremely Weak Text Supervision?

The paper explores the challenge of adapting Graph Neural Networks (GNNs) to learn from language with extremely weak text supervision. While contrastive learning models such as CLIP have shown success in vision-language alignment, GNNs struggle due to data scarcity, diverse task levels, and conceptual mismatches across domains. The authors introduce Morpher, a novel multi-modal prompt learning approach that aligns pre-trained GNN representations with Large Language Model (LLM) embeddings without fine-tuning either model. Morpher employs a graph-text multi-modal prompt learning paradigm, leveraging cross-modal projectors to map graph embeddings into the LLM’s semantic space. The paper uses network data in Chemistry and Biology to demonstrate the effectiveness of their method.

The paper offers several applications for social science research, particularly in network analysis, digital humanities, and computational sociology. In particular, if we were to consider data such as political discourse networks (e.g.,legislative voting records, citations in legal texts, or Twitter interactions among policymakers), we could be embedded the data in the same space as language models trained on political theory texts. This would enable semantic comparisons between network clusters and ideological constructs—helping researchers automatically classify emerging political alliances or detect shifts in ideological positioning, which provide a better platform to contextualize where the data is coming from.

For the goal above social network data (such as connections in political parties or social connections on social media) might be a good start, and use pre-trained LLM models familiar with the political landscape to support the task.

[youjiazhou]

https://arxiv.org/html/2402.17905v1

The article "Using Graph Neural Networks to Predict Local Culture" by Thiago H. Silva and Daniel Silver explores the application of GNNs to model and predict cultural attributes of urban neighborhoods. The authors address the challenge of understanding the dynamic and interconnected nature of neighborhoods by integrating multiple data sources, including demographic information, historical data, and movement patterns of groups between areas. Utilizing a large-scale dataset from Yelp, they demonstrate that GNNs can effectively capture the structural connectedness of neighborhoods, leading to accurate predictions of local cultural characteristics. Notably, their findings suggest that both local demographic data and group profiles derived from user reviews are equally effective in predicting cultural attributes. This highlights the potential of leveraging automatically extracted online data, especially in scenarios where traditional local information is scarce or unavailable.

The methodology presented in this study offers a valuable extension to social science research by providing a framework to quantitatively analyze the relational dynamics of neighborhoods. Traditional social science approaches often face limitations in capturing the complex interactions and temporal evolutions within urban settings due to data and methodological constraints. By employing GNNs, researchers can model neighborhoods as interconnected entities, allowing for the incorporation of both spatial and temporal data. This approach enables the examination of how movements of people, businesses, and ideas influence cultural and social transformations within urban areas.

To pilot this approach, one could utilize data from platforms like Yelp, which provide user-generated reviews and check-in information for various venues across different neighborhoods. This data can be enriched with demographic and socio-economic information from census databases to create a multifaceted representation of each neighborhood.

[Sam-SangJoonPark]

Exploring Auto-Encoders, Network Learning, and Table Learning in Social Science

For this week's "possibility" reading, I examined an article that explores how auto-encoders, network learning, and table learning can be applied to social science research. The paper highlights the use of auto-encoders for dimensionality reduction, table learning for structured data analysis, and network learning for uncovering relationships in complex systems like social networks or cultural shifts.

Key applications in social science include:

• Cultural evolution: Tracking how disruptions (e.g., travel, COVID-19) influence artistic innovation and consumer preferences.
• Political science: Identifying ideological shifts by embedding individuals or media sources in network-based models.
• Marketing and sociology: Modeling consumer behavior and social interactions to predict trends in digital engagement.

Potential data sources for applying these methods include:

• Spotify and music streaming datasets for studying how external disruptions shape musical exploration.
• World Values Survey (WVS) data to analyze ideological proximity and polarization.
• Reddit and Twitter networks to track discourse patterns and information diffusion.

By integrating network learning for relationships and table learning for structured data, these methods offer new perspectives on complex social behaviors.

[siyangwu1]

I have selected the paper titled "RecVAE: a New Variational Autoencoder for Top-N Recommendations with Implicit Feedback" for reflection.

Reflection:

The paper "RecVAE: a New Variational Autoencoder for Top-N Recommendations with Implicit Feedback" introduces an innovative approach to enhancing recommendation systems using variational autoencoders (VAEs). Traditional recommendation systems often struggle with effectively capturing user preferences, especially when dealing with implicit feedback data. This study addresses these challenges by proposing RecVAE, a model that incorporates several novel techniques to improve recommendation accuracy.

One of the standout features of RecVAE is the introduction of a composite prior distribution for the latent codes. This approach allows the model to better capture the complex patterns in user-item interactions, leading to more personalized recommendations. Additionally, the authors present a new method for setting the β hyperparameter within the β-VAE framework, which balances the trade-off between reconstruction accuracy and the capacity of the latent space. This careful tuning is crucial for modeling the nuances of user behavior.

The paper also details a unique training methodology based on alternating updates, which enhances the model's convergence and performance. Through extensive experiments on classical collaborative filtering datasets, RecVAE demonstrated significant improvements over existing autoencoder-based models, including Mult-VAE and RaCT. The comprehensive ablation study provided offers valuable insights into the contributions of each component of the proposed model.

In summary, this paper contributes to the field of recommendation systems by leveraging advanced VAE techniques to address the limitations of previous models. The proposed RecVAE model showcases the potential of combining deep learning architectures with probabilistic modeling to achieve state-of-the-art performance in top-N recommendations with implicit feedback.

https://arxiv.org/abs/1912.11160

[Daniela-miaut]

“Why do tree-based models still outperform deep learning on typical tabular data?” 2022. Grinsztajn, L., Oyallon, E., & Varoquaux, G. Advances in neural information processing systems, 35, 507-520.

This paper tries to understand why tree-based models perform better than deep learning on tabular data by first, establish a benchmark of 45 datasets for tabular learning and second, empirically investigate the different inductive biases of tree-based models and neural networks. The inductive biases tested and their results are: 1) when smoothing the target function, tree-based models’ performance decrease, which suggests that their performance are based on the inductive bias produced by a non-smooth target function, while neural networks would struggle to fit a non-smooth function; 2) by adding and removing uninformative features, the authors find that Multi-Layer Perceptron (MLP) like architectures are not robust to uninformative features; 3) another difference between tree-based models and neural networks is that in tree-based models are not rotationally invariant, which suits the properties of tabular data. The findings of this research might provide useful guidances for tabular deep-learning research.

I am curious how can further development in neural networks potentially make it achieve better performance than tree-based models, and what limitations of tree-based models can possibly be overcome by neural networks. It also seems to me intuitively that the structures of tree-based models and neural networks simulate different human cognitive functions, namely classification and understanding. While the former focuses on setting boundaries, the latter emphasizes finding learnable patterns in the data. I am curious about testing this hypothesis empirically. For example, we can find datasets that researchers traditionally used to do “understanding”, or pattern extraction tasks, such as text data, and transform them into tabular data and compare the performance of tree-based models and neural networks on these datasets, roughly in the same way provided by this paper. The results should help us understand different cognition patterns that people intuitively choose in different cognition tasks.

[shiyunc]

“Disrupted routines anticipate musical exploration Links to an external site.,” 2024. Kim, Khwan, Noah Askin, and James A. Evans. PNAS.

The study reveals how cultural tastes evolve as individuals’ experiences change, especially when daily life is disrupted. Analyzing millions of global music streaming plays, the study found that individuals’ music preferences diversified after traveling or experiencing events that disrupt daily routines, such as COVID-19 lockdowns. Travelers’ music choices align with the culture of the destination, while pandemic lockdowns have led people to prefer regional music rather than global average music preferences. The effects of this exploratory consumption are long-lasting, suggesting that music tastes reflect rather than compensate for changes in life, shaping long-term cultural consumption patterns.

Extending social science analysis:
I found the conclusion especially interesting that people's musical preference diverge instead of converge globally. We can compare the dynamics of musical exploration across different types of disruptions—do temporary disruptions like travel and long-term relocations result in different patterns of taste evolution? I think there are two explanations for musical taste change related to traveling: one is, as described in the paper, cultural exploration during the "unsettled lives/times", while the other might be the need to fit in a new environment. Exploring the cases of long-term relocation may help us clarify the mechanisms. By leveraging song2vec embeddings, we could quantify how closely an individual’s preferences align with the music traditions of their original residence (e.g., hometown) versus current residence. This study might explore the impact of immigration on global cultural trends.

Possible dataset:
The one used in the paper: the Deezer listener dataset (smaller user based, stronger localization)
Alternatively: Spotify Playlists dataset. We can retrieve it from Spotify official website :https://research.atspotify.com/datasets/

[xpan4869]

A Novel Deep Auto-Encoder Based Linguistics Clustering Model for Social Text

This paper presents a novel deep auto-encoder based approach for clustering short text in Urdu, a widely spoken but computationally under-resourced language. The authors created a standard dataset of 137,181 Urdu news headlines from multiple categories and developed a method that automatically learns feature representations while simultaneously determining cluster assignments. Their approach uses a deep neural network to transform high-dimensional textual features into a lower-dimensional space where clustering can be performed more effectively. The model iteratively refines cluster centroids by optimizing a KL divergence loss between soft assignments and an auxiliary target distribution. Through extensive experiments on Urdu news headlines, they demonstrated that their deep learning approach significantly outperforms traditional clustering methods like K-means, achieving 67.7% accuracy compared to 39.5% with standard methods. The work addresses the particular challenges of short text clustering, where limited contextual clues and sparse features make traditional clustering approaches less effective.

This deep auto-encoder based clustering approach could be extended to analyze cross-cultural communication patterns in multilingual social media environments. The method's ability to learn representations directly from text data without human-engineered features makes it particularly valuable for comparative analysis of discourse across languages with different structural properties. This could help detect emerging cross-cultural narratives, identify culturally-specific framing of global events, and trace how ideas evolve as they move between linguistic communities.

To pilot the deep auto-encoder based clustering model for analyzing cross-national narratives about LLMs, I would use a comprehensive multilingual dataset of public discourse collected from multiple sources. The dataset would comprise social media posts, news articles, and public comments about large language models gathered from five major language communities: English, Mandarin, Spanish, Arabic, and Japanese.

[CallinDai]

Sparse Auto-Encoder Interprets Linguistic Features in Large Language Models
https://arxiv.org/pdf/2502.20344

This paper explores how large language models (LLMs) encode and process linguistic knowledge by using Sparse Autoencoders (SAEs) to extract interpretable linguistic features. The study proposes SAELING, a framework that systematically decomposes LLMs’ hidden states into sparse high-dimensional representations, capturing distinct linguistic features across phonetics, phonology, morphology, syntax, semantics, and pragmatics. Through causal intervention experiments, the study introduces two key evaluation metrics—Feature Representation Confidence (FRC) and Feature Intervention Confidence (FIC)—which assess how well these extracted features correspond to actual linguistic structures and whether modifying them can steer model outputs. Their findings suggest that LLMs encode genuine structured linguistic knowledge and that targeted interventions can effectively manipulate model behavior.

I am also wondering what we can infer from their finding of the presence of inherent linguistic knowledge within LLMs?

Their study can advance cognitive science research by uncovering how linguistic features are encoded and processed in LLMs, offering insights into language comprehension and production. For instance, it can be used to examine how metaphor, syntax, or pragmatics emerge in neural representations, aligning with theories of cognitive architecture and linguistic cognition. By intervening in model activations, researchers can simulate controlled experiments to test hypotheses about how language structure influences meaning-making and decision-making.

A cognitive science pilot could analyze how metaphor processing differs between LLMs and human cognition. Using datasets of human-written metaphors (e.g., from literature, news, or experimental studies), we could extract metaphor-related features via SAELING and compare them to neural activation patterns in fMRI or EEG studies on metaphor comprehension. This approach would help evaluate whether LLMs represent metaphors in ways that align with human cognitive processing and identify key differences in abstraction and context integrations.