Visualization | Ibrahim Maïga

Global Warming Effects: Big Data Analysis Reveals Surprising Connections

Sat, 15 Jun 2024 00:00:00 +0000

1. Introduction

2. Data

3. ETL Pipeline

4. Database Migration

5. Dashboarding with Power BI

6. Results and Findings

7. Limitations and Challenges

8. Conclusion

9. References

Abstract

This report explores the impact of global warming on crime rates, birth rates, and energy consumption in the United States and Canada. With the evolution towards a digitally connected world, the volume, variety, and velocity of data have increased significantly. To derive valuable insights from this data, we utilized temperature, crime, natality, and energy consumption data from various sources. By employing big data tools and techniques, we transformed raw data into meaningful insights. Key findings indicate that higher temperatures are correlated with increased crime rates and energy consumption, while extreme temperatures are associated with reduced birth rates. This project demonstrates the potential of big data in enhancing our understanding of the implications of global warming.

This project aims to integrate and analyze diverse data sources to gain insights into the correlations between weather patterns, birth rates, crime statistics, and energy consumption in Canada and the USA. By leveraging big data technologies such as Hadoop, PostgreSQL, MongoDB, MySQL, and MSSQL Server, we aim to build a comprehensive ETL pipeline that facilitates the seamless collection, storage, and analysis of large datasets. The outcome will provide valuable insights to inform policy decisions and operational strategies across various sectors.

1. Introduction

The digital age has led to an exponential increase in data generation, characterized by high volume, variety, and velocity. Managing this much data requires robust data governance policies and best practices. However, the true value of data lies in its ability to provide actionable insights. This project aims to analyze the impact of global warming on crime rates, birth rates, and energy consumption in the US and Canada using temperature, crime, natality, and energy consumption data.

The goal of this project is to leverage big data technologies to process and analyze data from multiple sources, thereby gaining a better understanding of how rising global temperatures affect various societal metrics. By integrating and analyzing data from different databases, we aim to uncover patterns and correlations that can inform policy decisions and future research. The goal is to create a seamless data flow from collection to analysis, enabling comprehensive insights into how these variables interact over time and across regions.

2. Data

2.1 Data Sources

This project extracts data from various publicly available sources to collect the data required for our analysis, including:

APIs from weather services, crime statistics bureaus, and energy consumption reports.
Publicly available datasets from Government websites and international organisations.
Synthetic data.

In order to conduct our analysis, it is crucial to ensure that the datasets we collect are on a daily basis. This is imperative for our process. The datasets that were sourced in this project are listed below.

2.2 Databases

Data Collection and Preprocessing

Data was gathered from nine different sources, pre-processed, and stored in seven databases as follows:

Apache Hadoop: US crime data (us_crime.csv)
Snowflake: Canadian temperature data (ca_temp.csv)
PostgreSQL: US temperature data (us_temp.csv)
MySQL: US birth data (us_birth.csv)
SQL Server: US energy consumption data (us_energy.csv)
MongoDB: Canadian birth data (ca_birth.json)
Local Machine Storage: Canadian crime data (ca_crime.csv)

Initial data collection and preprocessing involved cleaning and standardizing the datasets. This included handling missing values, normalizing temperature readings, and ensuring consistency in the data formats across different sources.

For Canadian birth data, I could only get monthly counts, and I generated the daily counts using the actual monthly totals to ensure accuracy.

DBeaver UI on Azure VM

PROGRAMMING LANGUAGES

Python
SQL
PySpark
Shell Scripting

3. ETL Pipeline

DATA FLOW DIAGRAM

Our ETL process commences with extracting raw data from the data sources, which is then organised into several datasets. Each dataset undergoes a cleaning and pre-processing phase before being stored in its respective databases, such as Apache Hadoop, Microsoft SQL Server, PostgreSQL, and MongoDB. Additionally, one dataset is retained as a CSV file to replicate local data processing scenarios.

Apache Spark is utilised to consolidate data from different databases. The consolidated data is then exported to a CSV file and imported into Azure SQL Server using SSMS. Azure SQL Server, a cloud-based service, facilitates access to its data for all team members, regardless of location.

At the final stage, Power BI visualizes the data and provides insights for our research.

3.1 DATA CLEANING

Data Alignment:

Aligned datasets to a common timeline to ensure consistency in time series analysis. This involved transforming various temporal resolutions (daily, monthly, yearly) into a unified format.

Data Normalization:

Standardized units of measurement across different datasets (e.g., Fahrenheit to Celsius for temperature data
Harmonized categorical variables, especially in crime datasets, to ensure comparability between US and Canada data.

Handling Missing Data:

Identified and addressed missing values through interpolation or imputation methods.
Ensured that the missing data did not skew the results by applying appropriate statistical techniques.

Data Filtering:

Filtered out irrelevant data points and outliers that could affect the analysis. For instance, extreme temperature values that were likely recording errors were excluded.

Data Integration:

Merged datasets from different sources, ensuring data integrity and consistency.
Used ETL (Extract, Transform, Load) processes to consolidate data into a unified database, facilitating easier access and analysis.

Data Validation:

Validated the accuracy of the data by cross-referencing with reliable sources.
Ensured that the datasets were free from duplicates and errors.

3.2 DATA CONSOLIDATION

To facilitate comprehensive analysis, we configured an Azure VM with 4 CPUs and 8GB RAM to host a single-node Hadoop cluster, PySpark, and DBeaver. However, due to resource constraints, this VM was inadequate, leading to its replacement with a more powerful VM with 8 CPUs and 32GB RAM.

Using PySpark, we aggregated data from the six databases, performed additional cleaning, and joined the datasets to create a unified dataset. The combined dataset was then exported to Azure SQL Server and subsequently imported into PowerBI for analysis and reporting.

To effectively manage and analyze the large and varied datasets, we utilized PySpark for aggregation and querying. PySpark is the Python API for Apache Spark, which is a powerful open-source distributed computing system.

Benefits of Using PySpark:

Scalability: PySpark’s distributed computing capabilities allowed me to scale the data processing tasks across multiple nodes, handling large volumes of data efficiently.
Speed: Spark’s in-memory processing and optimized execution plans significantly reduced the time required for data aggregation and querying compared to traditional methods.
Integration: PySpark seamlessly integrates with various data sources, including HDFS, Apache Hive, and relational databases like MySQL and PostgreSQL, enabling easy data consolidation from diverse sources.
Flexibility: PySpark supports both SQL queries and complex data transformations using its DataFrame API, providing the flexibility needed for our data consolidation tasks.

Data Loading

I loaded the cleaned datasets into Spark DataFrames from various storage systems, including Snowflake and Hadoop Distributed File System (HDFS).

Data Transformation

Using PySpark’s DataFrame API, I performed necessary transformations such as filtering, joining, and grouping data. These transformations helped in aligning the data from different sources and preparing it for analysis.

Aggregation

PySpark’s powerful aggregation functions enabled me to compute summary statistics, such as average daily birth rates and total daily energy consumption, across large datasets efficiently.

Querying

For querying the data, I used PySpark SQL, which allowed me to run SQL-like queries on the dataframes. This approach provided an intuitive and familiar interface for data analysts to extract insights from the consolidated data.

Data Export

After performing the necessary aggregations and queries, the resulting datasets were exported to formats suitable for further analysis and visualization, such as CSV files and tables in a relational database. In our case we are using PowerBi for further visualization

4. Data Migration

Data migration involves transferring data from source systems to the target data warehouse. This process ensures that data is accurately and efficiently moved while maintaining its integrity. ETL scripting techniques and database connector tools facilitated smooth migration and data replication.

5. Dashboarding with Power BI

PowerBI was used to create various visualizations to represent the findings:

Heat maps depicting the correlation between temperature and crime rates.

Line graphs showing trends in birth rates relative to temperature variations.

Bar charts illustrating energy consumption patterns in relation to temperature changes.

Figure 1. The average temperature in the US vs Canada, in Celsius

Figure 2. Locations and quantities of data collected

Figure 3. Dashboard of US Statistics, including birth rate, energy demand, and crime rate

Figure 4. Dashboard of Canada’s Statistics, including birth rate, max and min temperatures, and crime rate

6. Results and Findings

Temperature Trends:

Average temperatures in the US are higher than in Canada. There is a noticeable increase in temperatures over the years, indicative of global warming trends.

Crime Rates:

Higher temperatures are positively correlated with increased crime rates in both the US and Canada. This correlation suggests that as temperatures rise, the likelihood of criminal activity also increases.

Birth Rates:

Extremely high temperatures are associated with reduced birth rates in both countries. This trend may be attributed to the adverse effects of heat on human health and fertility.

Energy Consumption:

Energy consumption is directly correlated with temperature variations. Higher temperatures lead to increased energy demand, particularly for cooling purposes, while lower temperatures increase heating needs.

7. Limitations and Challenges

Procuring datasets that matched for all the metrics we are querying

Datasets with dissimilar scopes of time, frequency of data collection, and even unit measurements were a significant challenge that necessitated the use of big data analytical tools to aggregate this data across multiple sources and adjust it to be comparable.
Ensuring that the crime data included a similar scope of criminality – certain crimes in Canada aren’t even considered crimes in the US.
Ensuring data relevant to my use case met the standards for big data in terms of size and format (CSV vs JSON).

Issues with resources and resource allocation

Handling large datasets requires substantial computational power and memory. My initial Azure Virtual Machine setup did not provide sufficient power to efficiently process and analyze the data, which led to multiple crashes of Spark sessions.
I encountered an issue with the Hadoop NameNode entering safe mode, which halted all read and write operations on the file system. Safe mode is generally enabled when the NameNode detects resource constraints or during startup until a minimum percentage of data blocks are replicated safely. This can disrupt workflows, especially in resource-intensive tasks.
Procuring high-quality datasets, especially from paid sources, and upgrading computational resources were constrained by budget limits.

Difficulties connecting different databases and aggregating data

Integrating data from heterogeneous sources (Hadoop HDFS, Snowflake, SQL Server, MongoDB, PostgreSQL, and MySQL) involved significant compatibility issues. Each database has its own query language and data handling mechanisms.
Aggregating data from different sources required complex ETL processes. Ensuring data integrity and consistency across databases was a major hurdle.
Querying large datasets across different databases resulted in high latency and performance issues, slowing down the analysis.
Each database have a different schema for storing similar data, necessitating a thorough understanding and mapping of these schemas for accurate data integration.

8. Conclusion

This project demonstrates the potential of big data in transforming raw data into valuable insights. By analyzing the impact of global warming on crime rates, birth rates, and energy consumption, I have uncovered significant correlations that can inform policy and decision-making. The findings highlight the importance of addressing climate change and its far-reaching effects on society.

Future work could involve extending the analysis to include additional metrics and regions, as well as exploring predictive modeling techniques to forecast the impacts of global warming. Continued advancements in big data technologies and methodologies will further enhance our ability to derive meaningful insights from complex datasets.

This project underscores the value of big data in understanding and addressing global challenges, reinforcing the need for robust data governance and analytics frameworks.

9. References

The Art and Science of Movie Production: A Data-Driven Approach

Thu, 11 Apr 2024 00:00:00 +0000

A. Definition, Objectives, and Scope

Company Overview

Bon Films Inc. is a mid-budget film production company, operating within a financial range of 100,000 to 1,000,000 USD. The company also runs an in-house streaming service that focuses on English and French-language films for the Canadian market. Unlike traditional film production companies that rely heavily on creative intuition, Bon Films employs a data-driven approach integrated into its production and distribution model. This method supports informed creative decisions, ensures efficient turnaround times, and maintains a balance between financial goals and artistic integrity.

The main objective is to use data insights to improve decision-making processes, especially in determining the optimal runtimes for movies and identifying the directors best suited for specific genres. This strategy aims to enhance the financial viability of projects while preserving the creative aspects of filmmaking.

Industry Context

The global entertainment and media industry is estimated to have a total revenue of $2.8 trillion USD in 2023, and is expected to grow at a compound annual growth rate of 3.9% over the next five years, as reported by PricewaterhouseCoopers. With the advent of streaming platforms, audience retention and engagement have become crucial metrics. Data-driven strategies are increasingly employed to mitigate risks and align productions with prevailing successful trends. Nonetheless, excessive reliance on data has often led to a saturation of repetitive content, potentially stifling creativity. By focusing on runtime and genre-specific data, Bon Films aims to produce content that balances creative uniqueness with commercial viability, emphasizing collaborations with emerging directors aligned with the company’s strategic goals.

Functional Areas of Focus

The analysis conducted in this context spans multiple functional areas, including finance, sales, and marketing. Financial investment strategies, marketing efforts, and production timelines are optimized through data analysis to align with audience preferences and industry standards. Understanding the correlation between movie runtime and genre is vital for producing content that meets both consumer expectations and financial objectives.

Objectives

The main objective is to develop a production strategy rooted in data-driven insights. This involves analyzing competitor data with a focus on key elements such as budget, revenue, runtime, cast, language, genre, and audience ratings. Such an analysis will inform strategic decisions for upcoming projects to maximize their potential for success, both at the box office and on the company’s streaming platform. Additionally, segmenting the analysis by genre ensures that movie formats align with audience expectations and minimize risks such as viewer drop-off due to excessive runtimes.

Another key objective is to identify emerging directors whose work aligns with Bon Films’ strategic vision and creative direction.

Text Table

Scope of Analysis

The scope of this analysis is confined to films that align with Bon Films’ budget range and have verifiable revenue data. Films with missing or incomplete revenue documentation will be excluded to ensure the results are relevant and applicable to the company’s operations and financial strategy.

B. Data Acquisition

The data for this analysis is sourced from The Movies Dataset on Kaggle, which consolidates information from TMDb and IMDb. These are well-established databases in the film industry, maintained and updated regularly by industry professionals. The dataset includes a unified identifier linking primary keys from both TMDb and IMDb, facilitating consistent data analysis. Preprocessing and data cleaning, such as JSON parsing for genre extraction, are conducted using Tableau Prep to ensure the dataset is ready for visualization and analysis.

C. Data Visualization

Data visualization is performed using Tableau, which enables the creation of comprehensive visual summaries. These include detailed text tables and interactive dashboards that highlight key trends and correlations in the data.

Dashboard

Data Discoveries

The challenge faced by Bon Films is selecting movie genres and optimal release schedules that maximize revenue. This requires a historical analysis of revenue trends, genre-specific profitability, seasonal performance, and return on investment (ROI).

Key Observations

Bon Films Inc. leverages data analysis and visualization to enhance its production processes. By using historical data and viewer preferences, Bon Films strategically optimizes projects to maximize revenue. Tools like Tableau Prep and Tableau are used to transform raw data into actionable insights, demonstrating technical expertise and facilitating complex data storytelling. This positions Bon Films as a proactive industry player employing innovative, data-driven strategies.

Release Timing: The analysis indicates that movie releases in June and December tend to yield higher revenue. This trend points to the potential benefits of strategically timing film releases to coincide with peak moviegoing periods, such as summer holidays and the festive season.

Genre Profitability: Action, Adventure, and Drama genres have demonstrated significant profitability. These genres have consistently attracted large audiences and generated substantial revenues, making them suitable for investment.

Viewer Preferences: Drama as a genre receives a high volume of viewer votes, indicating that audiences value it both critically and commercially. This suggests that producing quality Drama films could satisfy audience demand while maintaining profitability.

ROI Analysis: A positive correlation is evident between budget size and revenue, which is valuable for informing budget allocation decisions. Higher-budget productions tend to yield higher revenues, although this relationship must be balanced against potential financial risk.

Genre-Specific Revenue Insights: The following chart visualizes average budget and revenue by genre, highlighting that Animation, Adventure, and Family films command higher budgets and revenues, underscoring their profitability. However, these high-revenue genres risk saturation as they attract heavy investment, leading to increased competition and potentially diminishing returns. Conversely, genres like Romance, Comedy, and Horror, while yielding lower average revenues, present valuable growth opportunities with less competition, allowing Bon Films to diversify its portfolio strategically. This data-driven approach informs investment in promising genres while balancing potential risks in saturated markets.

Cluster Analysis

Trends and Patterns

Seasonal trends reveal that timing releases to align with peak viewing periods, like June and December, can greatly enhance box office performance. The Action and Adventure genres, in particular, demonstrate consistently high ROI, indicating their resilience to market fluctuations. Key insights from this analysis highlight optimal release periods, high-profit genres, and the positive relationship between budget and revenue. Recommendations include exploring genre crossovers, catering to audience preferences, and employing strategic budgeting to maximize returns.

Root Cause Analysis

The strategic goal of Bon Films is to use data to guide production decisions and mitigate risk without stifling creativity. Initial analyses reveal that uneven data distribution across genres, due to movies being tagged with multiple categories, can complicate genre-specific insights. To refine target genres, it is crucial to address this distribution issue.

Historical data also reveals that poorly reviewed movies often perform poorly at the box office. Thus, public reception and critical acclaim play a key role in determining commercial success. The hypothesis that shorter runtimes are more suitable for certain genres, such as Animation targeting younger audiences, finds support in the analysis. This allows for strategic adjustments to runtime, depending on the target demographic and genre.

Findings:

The historical revenue and budget trends chart for Bon Films indicates significant fluctuations over the years. Sharp peaks and valleys illustrate revenue volatility, with an average line providing a baseline for typical revenue and budget. The downward profit trend in 2023 warrants a focused investigation into genre choices and thematic content.

Budget vs Revenue

Runtime Considerations: High-performing Animation films often have longer runtimes and deviate from strictly family-oriented themes.

Action and Adventure: These genres benefit from extended runtimes, which allow for deeper story development and higher engagement.

Recommendation: For initial projects, producing an Animated Action-Adventure film approximately two hours in length and releasing it in December could align with observed revenue trends and audience expectations.

Insights and Recommendations

Scorecard Analysis:

Profit analysis from 2020 to 2022 shows a steady growth trajectory, indicating initial success in adopting data-driven production strategies. However, a profit decline of 23.1% in 2023 suggests the need for further investigation into specific areas such as genre choices, thematic content, and marketing strategies.

Budget and Revenue Observations:

The world map of film production budgets highlights the highest average film budgets in countries like Germany, China, and Australia. This insight helps Bon Films consider international market trends for potential expansion and strategic partnerships. While the U.S. and Canada have smaller average budgets, they offer unique production opportunities at competitive prices.

World Map

Strategic Recommendations:

Investigate Profit Declines: A thorough analysis of the 2023 decline should be conducted, focusing on genre distribution, thematic trends, and marketing effectiveness to identify actionable areas for improvement.

Budget Optimization: Refining budget allocation strategies to enhance efficiency without reducing production quality is essential for sustainable growth.

Adopt Proven Successful Elements: Studying high-performing films in the market for replicable success factors can offer valuable insights.

Innovative Content and New Directors: Introducing innovative themes and collaborating with emerging directors can infuse fresh perspectives and attract new audiences. Leveraging new technologies within Animation and other underutilized genres could further enhance Bon Films’ competitive position.

Utilizing a Pareto analysis of genre profitability can guide refined strategic decisions. By identifying top-performing genres responsible for the majority of revenue, resources can be optimally allocated to focus on these high-impact areas.

Pareto chart

D. Conclusion

Combining detailed data analysis with strategic cost management and creative innovation provides a clear pathway for Bon Films to achieve sustained growth and profitability. Aligning production strategies with data-driven insights ensures films resonate with audiences while meeting financial goals.

Predicting SpaceX Rocket Landings: A Data Science Journey

Sun, 21 Jan 2024 00:00:00 +0000

Introduction

SpaceX has redefined aerospace economics with its reusable rocket system, achieving historic milestones that set it apart in the space industry. Since December 2010, SpaceX has been the only private company to return a spacecraft from low-Earth orbit. In 2024, it offers Falcon 9 rocket launches at a highly competitive price of approximately 67 million dollars per launch, a significant reduction compared to other providers. The closest competitor to the Falcon 9 in terms of payload capacity and market segment is ULA’s Atlas V rocket, which had a launch cost of approximately 115 million per launch. The primary factor behind these cost savings is the reusability of the Falcon 9’s first stage, which allows for refurbishment and relaunch at a fraction of traditional costs.

Project Goal

The primary objective was to develop a machine learning model capable of predicting Falcon 9 landing successes. Such a model could inform launch cost estimations and enhance mission planning, offering additional insights that could benefit other space organizations aiming to optimize their own rocket landings.

Successful Landings

Failed landings

Data Collection and Initial Analysis

This project leverages a comprehensive dataset of SpaceX missions to explore the potential of machine learning in predicting Falcon 9 landing success, providing insights into the factors influencing landing rates, such as payload mass, orbit type, and launch site location. By analyzing these elements, machine learning models can support more accurate cost estimations, mission planning, and optimized launch operations, offering a deeper understanding of SpaceX’s competitive advantage in the space sector.

The analysis used launch data from two main sources:

Historical Data (2010-2020): Launch data from SpaceX’s history was obtained from existing datasets.
Recent Launches (2020-present): Supplemented with web scraping from Wikipedia using BeautifulSoup4.

Exploratory analysis identified several key patterns:

Launch Sites: Cape Canaveral Space Force Station (CCSFS) had the highest number of successful launches, while Kennedy Space Center (KSC) and Vandenberg Space Force Base (VSFB) showed the highest success rates at 77%.
Payload Mass: The highest success rates were observed in launches with payloads between 9,000-16,000 kg.
Booster Version: The Falcon 9 B5 version achieved a perfect 100% landing success rate.

Machine Learning Models and Performance

Five machine learning models were implemented and evaluated:

Logistic Regression Logistic Regression, used for binary classification, fits data to a logistic curve to predict event probabilities based on input features, modeling relationships between variables and binary outcomes.
Decision Tree A Decision Tree is a supervised learning algorithm used for classifying or predicting outcomes by branching data into subsets. It creates a tree-like model of decisions, where each node represents a feature and each leaf node signifies an outcome. This approach is valued for its clear interpretation and handling of categorical data.
Random Forest Random Forest is a popular machine learning algorithm known for its robustness and accuracy. It builds an ensemble of decision trees and combines their predictions to make more reliable classifications or predictions. This approach is effective for a wide range of tasks, making Random Forest a versatile choice in machine learning.
Stochastic Gradient Descent (SGD) Stochastic Gradient Descent (SGD) is a common optimization technique in machine learning. It updates a model’s parameters using random training examples, making it efficient for large datasets. SGD aims to minimize the loss function iteratively, making it useful for training machine learning models with complex optimization tasks.
Support Vector Machine (SVM) The Support Vector Machine (SVM) is a powerful machine learning algorithm utilized for classification and regression tasks. It is a supervised learning algorithm where a hyperplane in an n-dimensional space is found to maximally separate the different classes in the training data. Both linear and nonlinear classification tasks, as well as regression tasks, can be handled by SVMs. The algorithm is particularly useful when the number of features is large compared to the number of samples, and when the data is not linearly separable.

Each model underwent both default testing and GridSearchCV optimization. The Decision Tree model emerged as the top performer, achieving a 96.6% accuracy rate.

Model Comparison and Highlights

To see how the default models performed and how GridSearchCV affected different Models, let us summarize what we observed while working with these 5 models.

Logistic Regression

Default Model: Achieved the highest score amongst the other default models.
GridSearchCV: Did not substantially improve performance; default version outperformed it.

Random Forest

Default Model: Performed nicely with a high score.
GridSearchCV: Benefited from tuning, reaching higher best score and best estimator rating as compared to the default.

Decision Tree

Default Model: Performed reasonably well.
GridSearchCV: Showed improvement with higher best score, best estimator performance, and better confusion matrix performance.

SGD (Stochastic Gradient Descent)

Default Model: Showed decent performance.
GridSearchCV: Tuning improved the model, accomplishing higher best score and best estimator score.

SVM (Support Vector Machine)

Default Model: Showed incredibly lower overall performance.
GridSearchCV: Slightly improved best score but did no longer significantly affect the best estimator score or confusion matrix performance.

Overall

Logistic Regression: Reached 93% accuracy with default settings but faced challenges predicting failures accurately.
Random Forest: Demonstrated strong performance, with optimized accuracy reaching 93.1% and high precision in predicting successes.
Decision Tree: Excelled with 96.6% accuracy after optimization, marking it as the best-performing model.

Feature Selection and Insights

This feature selection task was performed to determine if there were any improvements with a reduced number of features. From the original dataset of 14 features, 5 features were removed, leaving the dataset with 9 features. The removed features were selected by a data preprocessing procedure with the 5 least important features removed. The reduced features dataset was then used to perform the 5 previously used models to determine how feature selection had impacted the accuracy scores.

How Features Were Selected for Removal?

Feature importance was assessed through data preprocessing, where the dataset was evaluated using various methods to identify the top nine features for each model. The counts of these selected features were tallied across all models, with the least frequently appearing features excluded from the analysis. This approach ensured a focus on the most impactful predictors for the task.

Several methods were used to identify critical features:

Pearson Correlation
Chi-Squared Analysis
Recursive Feature Elimination
Embedded Lasso Regularization
Embedded Random Forest and LightGBM

Overall, the accuracy was lower with the reduced feature dataset. This suggested that all the features were important for finding the best model. The only reason why feature removal would have been performed was to save space for storing the dataset or to reduce the model fitting computation.

The best non-tuned model was Linear Regression with a score of 0.93 for both datasets, the highest for untuned models. The best model for a tuned model was a decision tree with a score of almost 0.97 and 0.92 for the original and reduced datasets, respectively.

Interestingly, there was a 3% improvement between untuned and tuned hyperparameters for Random Forest of the reduced dataset that was not seen in the original dataset.

Precision improved with the reduced dataset for Linear Regression. Perhaps overfitting occurred with such a simple model with the original dataset. Decision Tree, Random Forest, and Stochastic Gradient Descent (SGD) all saw a reduction in precision with the reduced dataset, while Support Vector Machine (SVM) had no changes. These models needed more features to create an accurate model.

Recall overall stayed the same or reduced, with the exception of the Decision Tree. More features seemed to be more important for model fitting. The decision tree model might have been fitted with better features first, allowing a higher recall in the reduced feature dataset.

F1 scores increased for Linear Regression, Decision Tree, and SVM models but decreased for Random Forest and SGD models. There seemed to be no correlation between the original and reduced datasets.

Interactive Visualization of Launch Sites

The project utilized Folium to create interactive maps visualizing SpaceX launch sites, success rates, and proximities to possible launch trajectories. This approach provided geographical insights into launch success and suggested characteristics for optimal launch site locations.

Geographic Analysis Objectives

Marking Launch Sites: Visualizing each launch site on an interactive map.
Success and Failure Markers: Showing successful and failed launches at each site.
Distance Calculations: Calculating distances from each site to nearby locations, exploring proximities and their impact on success rates.

Challenges and Lessons Learned

One significant challenge involved handling imbalanced data, with successful landings outnumbering failures. This imbalance impacted the models’ ability to predict failures accurately despite high overall accuracy.

Practical Applications

The findings from this project can support:

Launch Cost Estimation: Landing success predictions enable more precise cost estimation.
Mission Planning and Risk Assessment: Accurate predictions aid in planning and risk management.
Resource Allocation: Success rates by site allow for optimized resource planning.

Future Improvements

Although the models performed well, additional improvements could enhance their predictive power:

Increasing Data Balance: Collecting more data on failed landings could improve prediction accuracy.
Adding Environmental Variables: Including weather data and other external conditions may improve predictions.
Developing Ensemble Methods: Combining models could yield even better results.
Incorporating Real-Time Data: Including telemetry data could enhance dynamic predictions.

Conclusion

This project underscores the potential of machine learning to address complex aerospace challenges. Predicting landing success can lead to more precise cost estimates and optimized launch site selection, both crucial for making space travel sustainable and economically viable.

This analysis highlights that simpler models, such as the Decision Tree, can sometimes outperform more complex ones when carefully optimized. As space exploration continues to evolve, data science will play a pivotal role in increasing the efficiency and accessibility of future space missions.