Travaux pratiques (TP) - Intelligence artificielle et données
29 documents à télécharger gratuitement
Travaux pratiques (TP) de intelligence artificielle et données partagés par des étudiants et des enseignants. Thèmes couverts : machine learning, apprentissage automatique, big data, data science, fouille de données.
This document explores unsupervised classification methods, focusing primarily on two types: hierarchical and non-hierarchical methods. It details the K-means algorithm, explaining its principles, convergence criteria, and the role of intra-class and inter-class variance. Hierarchical clustering is also discussed, describing dendrograms and aggregation criteria such as single linkage, complete linkage, and variance-based methods. Additionally, the document provides an overview of mixed classification, combining K-means partitioning with hierarchical clustering for a more refined analysis.
This document outlines methods of unsupervised classification, focusing on automatic classification techniques. It discusses the concepts of dissimilarity and inertia as they relate to grouping individuals based on their characteristics. Additionally, it presents the K-means algorithm for classifying data.
This document outlines a course workshop on data mining as part of the Business Analytics & Data Science master's program. It includes a section on principal component analysis (PCA) applied to vehicle data with specific tasks for analysis. The focus is on interpreting correlations and distributions of vehicles using PCA techniques.
The document outlines a practical workshop for Principal Component Analysis (PCA) applied to six continuous variables recorded from data on 20 cars. Using R programming and the FactoMineR and factoextra packages, correlations between variables are analyzed to identify patterns and influences in the factor space. The methodology includes generating a correlation matrix, interpreting factor axes, plotting correlation circles, and examining individual contributions. Results provide insights into the distribution of vehicles across the factor map, supported by graphical visualizations.
This document outlines an atelier focused on data mining within the Business Analytics & Data Science program for the academic year 2021-2022. It covers principal component analysis, correspondence analysis, and unsupervised classification methods. Additionally, it provides references to relevant resources such as cran.r-project.org and anaconda.com.
This document presents a data mining project focused on classifying road accidents in France for the year 2020. It includes data importation, variable selection, descriptive statistics, correlation analysis, PCA, and K-means clustering. The findings highlight the relationships among various factors contributing to road accidents.
Apache Spark is a powerful, open-source cluster computing platform designed for processing large-scale data. It provides high-level APIs in Java, Scala, Python, and R and supports batch and stream processing while leveraging in-memory computation for enhanced performance. Spark's layered architecture includes components like Spark SQL for structured data processing, Spark Streaming for real-time analytics, Spark MLlib for machine learning, and Spark GraphX for graph computations. Despite its advantages in processing speed and scalability, Spark has limitations such as higher memory costs, l...
This document introduces Apache Spark, a powerful open-source platform for cluster data processing. It covers the architecture, key components, and functionalities such as batch processing, stream processing, and machine learning capabilities. Apache Spark integrates well with Hadoop and provides a high-level API in multiple programming languages.
The document introduces the concept of Big Data by explaining its origins, characteristics, and challenges it imposes on traditional data management systems. It discusses the core 'Vs' of Big Data (Volume, Velocity, Variety) and their consequences on system architecture, such as the need for horizontal scalability, continuous availability, and flexibility. The text also covers the CAP theorem and its implications for distributed Big Data systems, highlighting choices between consistency, availability, and partition tolerance. Solutions like distributed and scalable infrastructures are neces...
This document introduces the concept of Big Data, highlighting its significance and the reasons behind its emergence. It discusses the need for new infrastructures to handle the vast amounts of data generated in today's digital age. Furthermore, it outlines the characteristics of Big Data, particularly focusing on the volume aspect and its implications.
This document discusses the impact of adjusting the 'max_depth' parameter in a machine learning model. It highlights the problem of overfitting and underfitting based on changes to this parameter and identifies the optimal value for achieving the best mean squared error. The findings include specific error values computed for test datasets and emphasize the application of grid search to optimize 'max_depth'.
This paper proposes a novel Feature Manipulation Defense mechanism to detect and purify adversarial examples in machine learning. By using a combo-variational autoencoder to manipulate semantic features, the method demonstrates high detection and purification accuracy across multiple datasets. The effectiveness of this approach lies in its ability to resist misclassification due to non-significant changes in input features.
The document provides a detailed hands-on exercise to optimize SSIS packages for incremental data extraction, employing methods like using a datetime column, Change Data Capture (CDC), CDC Control Tasks, and Change Tracking for efficient ETL operations. The first exercise demonstrates the use of datetime columns to extract modified records and implement the high water mark technique. The second exercise uses SQL Server's CDC feature to retrieve only modified rows from a database without utilizing a dedicated timestamp column, while also including a workflow for testing changes. Together, th...
This document outlines a lab exercise for optimizing SQL Server Integration Services (SSIS) packages to improve ETL processes by enabling incremental data extraction. The lab focuses on using datetime columns, Change Data Capture (CDC), and Change Tracking mechanisms to filter out previously extracted data, reducing system load and resource consumption. The steps include configuring SSIS controls, SQL queries, and variables to implement the 'high-water mark' technique. Additionally, the lab demonstrates enabling SQL Server's CDC feature and creating a stored procedure to retrieve modified r...
This document explores the application of regular expressions in R for processing email data. Initially, challenges in parsing email domains are identified, leading to the utilization of specific regular expressions to extract data systematically. Functions like 'grep', 'gsub', 'regexpr', and 'regmatches' are discussed and applied to manipulate, search, and modify email strings effectively. Results include generating a frequency table for email domains from the processed dataset.
The document provides an overview of methods to import, read, and manipulate tabular data formats in R, such as CSV and Excel files. It discusses the use of built-in functions like `read.csv` and libraries like `readxl` for handling external datasets. Methods to access specific rows, columns, and subsets of these data frames are thoroughly explained. Additionally, the document introduces built-in datasets in R and demonstrates their usage with key functions such as `data()` and `help()`.
This document outlines the implementation of data warehouse schemas for Adventure Works Cycles, enabling reporting and analytical capabilities for key business metrics. The lab covers designing, creating, and implementing both star and snowflake schemas, including relevant fact and dimension tables. A time dimension table is also created and populated to facilitate time-based analysis of business measures. The resultant schema supports normalized tables, hierarchical relationships, and integrates dimensions like geography and time for enhanced querying and reporting.
This document provides a practical introduction to data warehousing, using a fictional company called Adventure Works Cycles as a scenario. Key tasks include exploring various business data sources, running ETL (Extract-Transform-Load) processes using SQL Server Integration Services, and querying the resultant data warehouse for business insights. The methodology emphasizes staging data, cleansing processes, and centralized storage to solve distributed data challenges. Findings highlight the ability to consolidate data for improved reporting and analysis across multiple business domains.
This document describes a series of exercises focused on generating and interpreting outputs using SPSS statistical software. Tasks include creating frequency tables and cross-tabulations for variables such as travel purpose, global satisfaction, and respondent demographics (e.g., gender, profession). Graphs such as histograms and pie charts are created to visualize data, and analytical insights involve calculating age statistics and average scores for key criteria (ticket price, service quality, and flight punctuality). The exercises emphasize cross-tabulation with percentages for deeper i...
This lab focuses on developing an algorithm to classify images of different shapes (hearts, clubs, diamonds, spades) using various features such as central and Hu moments. It includes steps for segmentation, data selection, feature extraction, and testing the model using a confusion matrix. The goal is to improve the model's accuracy by selecting relevant features.
This document explains the implementation of a word count program using the MapReduce programming paradigm in a Hadoop framework. It describes the implementation of the Driver, Mapper, and Reducer classes and their specific roles in the program. The code processes input data by tokenizing words in a text file and counting the occurrences of each. Results are then written to an output path, which the user can specify during execution.
This lab involves implementing control flows in SQL Server Integration Services (SSIS) to manage ETL processes for Adventure Works Cycles. The exercises cover tasks like using tasks and precedence constraints, creating parameters and variables, and utilizing containers. Methodologies include setting up control flow, sending notification emails for task status, dynamically generating file paths using variables and parameters, and grouping tasks within sequence containers. The findings showcase the successful implementation of automation and error handling in ETL workflows, including the use...
The document details the procedure for creating an ETL solution using SQL Server Integration Services (SSIS). It involves extracting customer and sales data from the InternetSales database and loading it into a staging database with calculated sales amounts. Data profiling methods are utilized to examine column statistics, null ratios, and column lengths. The lab walks through creating and executing data flows, setting up transformations, and saving outputs to validated flat files or staging databases.
The lab focuses on creating an ETL solution with SQL Server Integration Services (SSIS). Users will extract, transform, and load customer and sales order data from various databases to a staging database. Methodologies include data profiling, customization of data flow tasks, transformations, and error handling during tasks using Visual Studio and SQL Server tools. The report emphasizes careful source data analysis, efficient ETL task creation, and data validation to meet data pipeline requirements.