Notebooks on Biraj Koirala

A Comprehensive Analysis of Image Captioning Models - Evaluating ViT-GPT2, BLIP, and GIT

Tue, 10 Dec 2024 00:00:00 +0000

This project is a comparative study of Image caption generation model . This experiment aims to provide:

A detailed breakdown of the architectures and mechanisms of ViT-GPT2, BLIP, and GIT.
A quantitative and qualitative analysis of their performance.
Insights into the strengths, limitations, and suitability of each model for real-world applications.

The dataset utilized for this study consists of 600 images, sourced exclusively from open-access platforms to ensure accessibility and reproducibility. Each image was meticulously self-annotated with high-quality captions to create a reliable ground truth for evaluating the models’ performance.

Visit the Project on GitHub View data in Kaggle

Dataset Composition

Animals - Includes various species in diverse settings, such as wildlife, pets, and zoos.
Humans - Depicts people in natural environments, performing activities, and interacting with objects.
Architecture - Captures man-made structures, including buildings, bridges, and urban landscapes.
Natural Formations and Nature - Covers landscapes, forests, mountains, rivers, and other natural scenes.
Everyday Objects - Features commonly found objects, such as tools, household items, and vehicles.

The data that we collected have been uploaded to Kaggle. Please check them out here.

Dataset Challenges

Diversity of Visual Content: Ensuring the dataset captures a wide variety of visual scenes and objects for generalizability.
Annotation Quality: Maintaining consistency in style and accuracy across all annotated captions.
Ambiguity: Handling images with multiple possible interpretations, where different valid captions could describe the same image.

Sample Image and Prompt

Custom Annotation - Man attempting a slam dunk
vit-gpt2 - a woman jumping in the air to catch a frisbee
blip-conditional - a photography of a basketball player jumping to the basket
blip-unconditional - a man jumping in the air with a basketball
git - a young man playing basketball in a gym

Models used for caption generation

The models that were used for caption generation are:

The folder captions in the repository contains all groundtruth captions as well as the model generated captions.

The annotations can be viewed in this sheet or in all_captions.csv file.

We used three metrics for our comparative study:

METEOR
BLEU-1
BLEU-2

Results

The results are calculated in score.ipynb notebook.The table and the graphs obtained from the study is shown below:

Model	METEOR	BLEU-1	BLEU-2
ViT-GPT2	0.1644	0.1845	0.0816
GIT	0.2207	0.2301	0.117
BLIP (Conditional)	0.2418	0.2357	0.1246
BLIP (Unconditional)	0.2426	0.2555	0.1327

Quantitative Results and Qualitative Analysis

The quantitative results are :

BLIP (Unconditional Mode) achieved the highest scores across all metrics (METEOR: 0.2426, BLEU-1: 0.2555, BLEU-2: 0.1327).
BLIP (Conditional Mode) closely followed, showing slight improvements in guided captioning (METEOR: 0.2418, BLEU-1: 0.2357, BLEU-2: 0.1246).
GIT demonstrated a balanced performance (METEOR: 0.2207, BLEU-1: 0.2301, BLEU-2: 0.1170).
ViT-GPT2 performed the weakest, struggling with visual-text alignment (METEOR: 0.1644, BLEU-1: 0.1845, BLEU-2: 0.0816).

The qualitative analysis that we made are:

BLIP models generated semantically rich and contextually accurate captions.
- GIT provided coherent but sometimes generic captions.
- ViT-GPT2 struggled with misidentification and irrelevant outputs.

Model Strengths and Weakness

The strength are as follows:

BLIP’s Dual Mode (Conditional/Unconditional) allowed better flexibility in caption generation.
GIT’s unified transformer architecture helped in balancing vision-language processing.
ViT-GPT2’s modularity enabled adaptability in vision and text alignment.

The weakness are as follows:

BLIP required significant computational resources.
GIT lacked interpretability due to its tightly coupled vision-language representation.
ViT-GPT2 frequently misidentified objects and actions, leading to less reliable captions.

Evaluation Metrics

- METEOR captured semantic accuracy.
- BLEU-1 and BLEU-2 measured word precision and phrase coherence.
- Other advanced metrics (CIDEr, ROUGE-L, SPICE) were not included, limiting evaluation depth.

Limitations

- Small dataset size (600 images) reduced statistical reliability.
- Lack of advanced evaluation metrics affected a deeper analysis.
- Real-world applicability was not tested, limiting practical insights.

Combined METEOR for models tested

Combined BLEU-1 for models tested

Combined BLEU-2 for models tested

Visit the Project on GitHub View data in Kaggle

Has Nepal been a gerontocracy?

Mon, 22 Nov 2021 00:00:00 +0000

Nepal’s political landscape has seen ongoing turmoil, from the dissolution of parliament quashed by the Supreme Court to persistent political unrest. In the midst of this chaos, I was curious to understand how Nepal reached this state. My journey began with an attempt to learn more about the country’s politicians by exploring the official website of the Prime Minister of Nepal. Unfortunately, the website provided little information. Determined to dig deeper, I explored various government websites but still found no comprehensive data about Nepalese politicians.

Finally, I stumbled upon a Wikipedia page that listed notable politicians in Nepal. It wasn’t perfect, but it offered a starting point. By scraping data from this page, I was able to compile a dataset of Nepalese politicians’ ages.

The final list only accounts for the politicians that are alive. Also, the list fails to capture all of the politicians. I was able to retrieve information of a mere 54 politician.

Key Findings from the Dataset

Despite its limitations, the dataset reveals some important insights:

Dominance of Older Age Group : The majority of politicians fall within the age range of 60 to 80 years, highlighting the prevalence of senior citizens in Nepal’s political landscape.
Average Age : The average age of the politicians in the dataset is approximately 63 years, classifying most of them as senior citizens under Nepal’s Senior Citizen Act (60+ years).
Lack of Youth Representation : Politicians aged below 40 years are extremely rare. Only two politicians in the dataset are under the age of 45, with the youngest being just 24 years old.
Significant Age Gap : The gap between the youngest politician (24 years) and the oldest politician (92 years) is a striking 68 years, indicating a lack of generational diversity in Nepalese politics.

Interestingly, Nepal’s Senior Citizen Act (2006) defines a senior citizen as anyone aged 60 or older. This means that the average age of Nepalese politicians falls squarely into the senior citizen category, confirming that Nepal operates as a gerontocracy—rule by the elderly.

Nepal vs. Global Trends

The Global Parliamentary Report by UNDP offers the following benchmarks:

→ The global average age of Members of Parliament (MPs) is 53.
→ The average age of women MPs is 50.
→ Sub-Saharan African MPs have the youngest average at 49, while Arab countries have the oldest average at 55.

Nepal’s average political age of 63 exceeds the global norm, making it one of the oldest among nations. This disparity highlights significant barriers for younger generations to enter politics.

Barriers to Youth Participation

The dataset underscores the difficulty Nepali youth face in pursuing political careers. The political system is heavily skewed toward older generations, who dominate decision-making roles. For a more dynamic and inclusive political environment, the older generation must mentor and empower their successors, ensuring a smooth generational transition.

The sample data is presented below. For the complete source code please visit Github . You can get the raw data here .

Last scraped : 22/11/2021

Name	Date of Birth	Age
Dilendra Prasad Badu	1954-01-09	66
Khadgajeet Baral	1928-04-17	92
Dev Raj Bhar	1952-01-29	68
Lekh Raj Bhatta	1960-03-24	60
Ranju Darshana	1996-03-03	24
Arzu Rana Deuba	1962-01-26	58
Sher Bahadur Deuba	1946-09-12	74
Bam Dev Gautam	1948-07-09	72
Kul Bahadur Gurung	1935-11-23	84
Tek Bahadur Gurung	1954-11-26	65
Ram Karki	1956-10-16	64
Sushila Karki	1952-06-07	68
Narayan Khadka	1949-03-20	71
Jhala Nath Khanal	1950-05-20	70
Shekhar Koirala	1950-08-25	70
Krishna Bahadur Mahara	1958-06-29	62
Chandra Maharjan	1954-11-16	65
Ram Saran Mahat	1951-01-01	69
Rajendra Mahato	1958-11-19	61
Radha Krishna Mainali	1946-09-26	74
Salma Khatoon Mikrani	1976-01-25	44
Madhav Kumar Nepal	1953-03-06	67
KP Sharma Oli	1952-02-22	68
Bal Chandra Poudel	1959-06-22	61
Binda Pandey	1966-12-30	53
Surendra Pandey	1958-08-25	62
Tilak Pariyar	1943-10-31	76
Ananta Prasad Paudel	1962-07-28	58
Bishnu Prasad Paudel	1959-11-20	60
Shankar Pokharel	1963-02-27	57
Sher Dhan Rai	1971-02-23	49
Prakash Man Singh	1956-04-03	64
Dal Bahadur Sunar	1959-10-01	61
Surya Man Dong Tamang	1970-04-20	50
Gagan Thapa	1976-07-16	44
Hridayesh Tripathi	1959-06-18	61
Amod Prasad Upadhyay	1936-03-29	84
Ram Baran Yadav	1948-02-04	72
Upendra Yadav	1960-12-11	59

Conclusion

The data reveals that Nepal’s political system is overwhelmingly dominated by senior citizens, significantly surpassing global age averages for politicians. While experience and wisdom are essential in leadership, it is equally important to encourage youth participation to bring fresh perspectives and ensure long-term political stability. The old guard must take proactive steps to mentor and pave the way for the next generation of leaders.

Nepal’s path to a more inclusive and dynamic political system lies in its ability to balance tradition with innovation—bridging the generational gap in governance.

Visit the Project on GitHub