Spark df profiling pypi ); Warnings: A summary of the problems/challenges in the data that you might need to work on (missing data, inaccuracies, skewness, etc. For each column the following statistics - if relevant for the column type - are presented in an interactive HTML report: Mar 15, 2022 · Download files. 2. Sweetviz is an open-source Python library that generates beautiful, high-density visualizations to kickstart EDA (Exploratory Data Analysis) with just two lines of code. For small datasets, these computations can be performed in quasi real-time. 1 dbutils. ) PySpark Integration#. Feb 17, 2023 · Data profiling is known to be a core step in the process of building quality data flows that impact business in a positive manner. (Dependencies are only required when explicitly requested. Project Description Feb 21, 2024 · 要继续对数据进行分析，请使用 ydata-profiling！该存储库实现了在 PyPI 上停用 pandas-profiling 软件包的减负策略。 随着pandas-profiling 的发展，有一个新的令人兴奋的功能 - 从版本 4. 1 - a Python package on PyPI - Libraries. Mar 10, 2022 · ⚠️ Warning ⚠️: The outputs of an H3 join are approximate – all resulting geometry pairs should be considered intersection candidates rather than definitely intersecting. Dependency Tree for spark-df-profiling-optimus 0. Jul 27, 2022 · Hashes for pyspark-dbscan-1. This will help in profiling data. 5. Start using Socket to analyze mars-profiling and its dependencies Dec 16, 2020 · pyspark-flame. Hashes for Spark-df-Cleaner-0. e. The autoreload instruction reloads modules automatically before code execution, which is helpful for the update below. html") # Will generate the report into a html file Mar 25, 2025 · Like pandas df. It offers functionalities for both univariate, bivariate analysis and multivariate analysis, handling missing values, outliers, and visualizing data distributions. You switched accounts on another tab or window. Details for the file spark-profiling-0. The pandas df. types import DecimalType, DateType, TimestampType, IntegerType, DoubleType, StringType from ydata_profiling import ProfileReport def profile_spark_dataframe (df, table_name ): """ Profiles a Spark DataFrame Nov 14, 2023 · DataProfileViewerAKP. The data can be verified based on the predefined data quality constraints. tar. Details for the file pydeequalb-0. Details for the file spark_profiling-0. Diff: A diff transformation and application for Datasets that computes the differences between two datasets, i. @julioasotodv / Latest release: 1. This project provides extensions to the Apache Spark project in Scala and Python:. Java >= 1. Here we will read the file directly from our GitHub repository. io Mar 6, 2024 · Hashes for pyspark_connectby-1. Batch and streaming support: Use DQX with your Spark pipelines. The significance of the package lies in how it Profiling large datasets. init_db (clear = True) # profile the historical data, register the dataset in the Metrics Repository and # optimize ML models for all profiling time series. 1 on Pypi Generating dependency tree Libraries. What's SourceRank used for? SourceRank is the score for a package based on a number of metrics, it's used across the site to boost high quality packages. run() # printing all the columns and their corresponding profiled data. th. 10, and installed using pip install spark-df-profiling in Databricks (Spark 2. These statistical summaries of datasets are commonly referred to as data "profiles" and capture the key information about the distributions of data within those datasets. profile_report(style={‘full_width The pandas df. spark_to_polars (spark_df) # Polars to Spark spark_df = converter. sql. ️ author: Mitchell Lisle. spark-df-profiling - Python Package Health Analysis | Snyk PyPI Create HTML profiling reports from Apache Spark DataFrames - 0. fixture ('fake_insurance_data. csv",header = True) # Use the custom extension Jul 27, 2022 · Hashes for pyspark-dbscan-1. For each column the following statistics - if relevant for the column type - are presented in an interactive HTML report: Apr 26, 2020 · Generate profile report for spark DataFrame. Prerequisites. 5 0. Hi! Perhaps you’re already feeling confident with our library, but you really wish there was an easy way to plug our profiling into your existing PySpark jobs. Generate profile report for Mars DataFrame. It is based on pandas_profiling, but for Spark's DataFrames instead of pandas'. File metadata Mar 14, 2025 · RayDP: Distributed Data Processing on Ray. polars_to_spark (polars_df Jan 4, 2023 · import thoth as th # init the Metrics Repository database th. The primary inspiration for this project was quickly comparing two datasets from a number of different formats after some transformation was applied, but a range of capabilities have/will continue to been implemented. read_csv (resources. For larger datasets, deciding upfront which calculations to make might be required. import spark_df_profiling. It is the first step — and without a doubt, the most important Jul 26, 2016 · Generates profile reports from an Apache Spark DataFrame. 12: September 6th, 2016 16:24 YData-profiling is a leading tool in the data understanding step of the data science workflow as a pioneering Python package. whylogs profiles are descriptive, lightweight, and mergeable, which makes them the perfect Dec 24, 2023 · # Profiling Data from pydeequ. profiles. 2 (2016-07-26) Jun 2, 2022 · Photo by Joshua Sortino on Unsplash. onData(yellow_df) \ . Set criticality levels: Quarantine or mark invalid data based on severity. PyDeequ is a Python API for Deequ, a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets. PyPMML-Spark is a Python PMML scoring library for PySpark as SparkML Transformer, it really is the Python API for PMML4S-Spark. You can find an example of the integration here. describe() function is great but a little basic for serious exploratory data analysis. Create HTML profiling reports from Apache Spark DataFrames - 0. Jan 12, 2024 · Data Verification. March 2023: You can now use AWS Glue Data Quality to measure and manage the quality of your data. show_notebook() # to show in a notebook cell my_report. For each column the following statistics - if relevant for the column type - are presented in an interactive HTML report: Jan 7, 2024 · Components of whylogs. 1 was published by pyodps. 10. :. Examining the data to gain insights, such as completeness, accuracy, consistency, and uniqueness. 0 onwards Data testing, monitoring, and profiling for Spark Dataframes. formatters as formatters, spark_df_profiling. Spark Extension. ) Feb 27, 2024 · Data profiling is analyzing a dataset's quality, structure, and content. The size of the example DataFrame is very small, so the order of real-life examples can be altered with respect to the small example. . dq_ob. profiles import ColumnProfilerRunner # Profiling all the columns: ColumnProfilerRunner. pip install pandas-profiling The pandas_profiling library in Python includes a method named as ProfileReport() which generates a basic report on the input DataFrame. option ("header", "true"). ) spark-df-profiling Releases 1. Feb 27, 2024 · Data profiling is analyzing a dataset's quality, structure, and content. soda. builder. Create HTML profiling reports from Apache Spark DataFrames. appName ("example"). The usage of the DataFrame is the same with PySpark. RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries, making it simple to build distributed data and AI pipeline in a single python program. Missing values analysis; Interactions; Improved histogram computation; Profiling with Spark DataFrames Jun 8, 2023 · Option 1: If the spark dataframe is not to big you can try using a pandas profiling library like sweetviz, e. Recent updates to the Python Package Index for spark-df-profiling Create HTML profiling reports from Apache Spark DataFrames. apply_data_profiling (source_config_df = config_df, write_consolidated_report = True) # Generating a data profiling report as well as recommending the quality rules based on the profiling report. faker-pyspark provides PySpark based fake data for testing purposes. from pytest import fixture from pysparkdt import spark_base @fixture (scope = 'module') def spark (): yield from spark_base (METASTORE_DIR) Metastore Initialization: Use reinit_local_metastore Apr 3, 2024 · By calling . Version: 0. Current version has following attributes which are returned as result set: Jan 30, 2023 · ydata-profiling primary goal is to provide a one-line Exploratory Data Analysis (EDA) experience in a consistent and fast solution. Reload to refresh your session. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. You need to run this one-liner to profile the whole dataset in one shot. # You can read data from all the supported sources as per Apache Spark module df = spark. csv ("<your path>") 3. Generates profile reports from a pandas DataFrame. tempo - Time Series Utilities for Data Teams Using Databricks. If you're not sure which to choose, learn more about installing packages. Documentation | Slack | Stack Overflow. This class contains the Then, using ydata-profiling is a simple two-step process: Create a ProfileReport object using one of: analyze(), compare() or compare_intra() Use a to_notebook_iframe() function to render the report. AutoViz('hcc. sql import SparkSession from duckdb_extension import register_duckdb_extension spark = SparkSession. spark-df-profiling-new Releases 1. This package provides a way to convert protobuf messages into pyspark dataframes and vice versa using pyspark udfs. 1. polars_to_spark (polars_df Feb 2, 2015 · Optimus is the missing framework for cleaning and pre-processing data in a distributed fashion with pyspark. xlsx files from Azure Blob storage into a Spark DF. gz; Algorithm Hash digest; SHA256: 9fcd8ed68f65aca20aa923f494a461e0ae64f180ee75b185db0f498a58b2b6e3: Copy : MD5 # Generating a data profiling report. templates as templates from matplotlib import pyplot as plt from pkg_resources import resource_filename Dec 19, 2024 · Spark Fixture: Define fixture for the local spark session using spark_base function from the testing package. Installation (pip): In your terminal just type pip install optimuspyspark Nov 30, 2022 · Data Comparator Overview. sparkpolars is a lightweight library designed for seamless conversions between Apache Spark and Polars without unnecessary dependencies. To install: pip install pbspark Usage Remark: Spark is intended to work on Big Data - distributed computing. Features supported: Univariate variables' analysis; Head and Tail dataset sample; Correlation matrices: Pearson and Spearman; Coming soon. PyDeequ is written to support usage of Deequ in Pyth Overview. However, you can first download the file and then upload it to your working directory and use it as AutoViz_Class(). 3. io soda-core-spark-df==3. set_data_source_name ("spark_df") # Attach a Spark session scan. By default, ydata-profiling comprehensively summarizes the input dataset in a way that gives the most insights for data analysis. In-depth EDA (target analysis, comparison, feature analysis, correlation) in two lines of code!. Let's get started and import ydata-profiling, pandas, and the HCC dataset, which we will use for Aug 4, 2015 · spark-df-profiling Create HTML profiling reports from Apache Spark DataFrames @julioasotodv / Latest release: 1. File metadata Jun 7, 2023 · pbspark. describe() function, that is so handy, ydata-profiling delivers an extended analysis of a DataFrame while allowing the data analysis to be exported in different formats such as html and json. Pandas Profiling can be easily installed using the following command. 13-py2. SparkDantic. , pandas, etc. faker-pyspark is a PySpark DataFrame and Schema (StructType) provider for the Faker Python package. YData-profiling is a leading tool in the data understanding step of the data science workflow as a pioneering Python package. phik_matrix # get Jul 26, 2016 · Generates profile reports from an Apache Spark DataFrame. show_html(filepath="report. pip3 install spark-df-profiling Feb 8, 2023 · Download files. File metadata Feb 27, 2025 · Apache Spark. getOrCreate # Create converter converter = DataFrameConverter # Spark to Polars polars_df = converter. It takes English instructions and compile them into PySpark objects like DataFrames. 0rc9. Dec 7, 2021 · File details. Features 1. Apr 20, 2016 · spark-df-profiling Create HTML profiling reports from Apache Spark DataFrames. Feast is the fastest path to manage existing infrastructure to productionize analytic data for model training and online inference. gz. restartPython() 2. rules_config = dq_ob. A Python API for Intelligent Visual Discovery. Apr 26, 2020 · File details. 导航. 14: May 27th, 2021 22:17 Subscribe to an RSS feed of spark-df-profiling-new releases Libraries. Mar 1, 2024 · Among the many features that PySpark offers for distributed data processing, User-Defined Functions (UDFs) stand out as a powerful tool for data transformation and analysis. Jun 22, 2023 · PySpark provider for Faker. This is a spark compatible library. Type casting between PySpark and pandas API on Spark; Type casting between pandas and pandas API on Spark; Internal type mapping; Type Hints in Pandas API on Spark. Чтобы установить модуль, напишите в PyDeequ . Jun 2, 2024 · pip install -i https://pypi. read. Mar 15, 2023 · dq-module is a tool which can be used to perform validations and profiling on the datasets. gz')) df. When viewing the contents of a data frame using the Databricks display function (AWS|Azure|Google) or the results of a SQL query, users will see a “Data Profile” tab to the right of the “Table” tab in the cell output. May 12, 2025 · A pydantic -> spark schema library. The significance of the package lies in how it What's SourceRank used for? SourceRank is the score for a package based on a number of metrics, it's used across the site to boost high quality packages. describe()函数一样方便，ydata-profiling对DataFrame进行全面分析，并允许将数据分析导出为不同的格式，如HTML和JSON。 pandas_on_spark. A low-overhead profiler for Spark on Python. whl; Algorithm Hash digest; SHA256: ecaedec3b3e0a2aef95498f27d64d7c2fabbc962a54599a645cf36757f95078b See full list on libraries. 1: September 4th, 2017 21:04 Browse source on GitHub View diff between 0. 4 - a Python package on PyPI - Libraries. Aug 29, 2024 · Welcome to the documentation for SparkKG-ML, a Python library designed to facilitate machine learning with Spark on semantic web and knowledge graph data. PyDeequ is a Python API for Deequ, a library built on top of Apache Spark for defining “unit tests for data”, which measure data quality in large datasets. read. py3-none-any. pandas_profiling extends the pandas DataFrame with df. It still seems like the wild west of Data Quality these days. 0 / ( 1) 废弃 'pandas-profiling' 包，请使用 'ydata-profiling' 代替. Awesome spark_jdbc_profiler created by hgbink - 1. Download the file for your platform. 7. Tools like Apache Deque are just too much for most folks, and Data Quality is still new enough to the scene as a serious thought topic that most tools haven’t matured that much, and companies dropping money on some tool is still a little suspect. It also computes the Kolmogorov-Smirnov test statistic to measure the distribution difference for numeric columns with low cardinality. You can define your checks in-line in the notebook, or define them ydata-profiling primary goal is to provide a one-line Exploratory Data Analysis (EDA) experience in a consistent and fast solution. This library contains a SingleDatasetQualityCheck() class which can used to validate the dataset against a defined set of rules. Pages for logged out editors learn more. You signed out in another tab or window. 8; Python 2. 13: September 6th, 2016 16:52 Browse source on GitHub View diff between 1. csv'). for col_name, profile in result. 13 and 1. 7-py3-none-any. add_spark_session (spark) # Define checks for datasets # A Soda Check is a test that Soda Library performs when it scans a dataset in your data source. Create a config in the form of python dict or read it from any json file spark-df-profiling. Jan 9, 2024 · import pandas as pd import phik from phik import resources, report # open fake car insurance data df = pd. This tool is compatible with two run_engines pyspark and polars. df() your table data will be read as Spark's DataFrame. Create Data & Add Soda CL Checks. 0. Data Validation. profile_create_optimize (df = history_df, # all your historical data dataset_uri = "temperatures", # identification for the Feb 2, 2015 · Optimus is the missing framework for cleaning and pre-processing data in a distributed fashion with pyspark. The English SDK for Apache Spark is an extremely simple yet powerful tool. getOrCreate # Register the DuckDB extension register_duckdb_extension (spark) df = spark. ) Sep 29, 2019 · Pandas-Profiling pip install pandas-profiling import pandas_profiling. whl. io. Mar 30, 2022 · A Python API for Intelligent Data Discovery. Pandas Profiling. appName ("DuckDB Example"). 0) I am able to import the module, but when I pass a data You signed in with another tab or window. The WhyLabs Platform relies on statistical summaries generated by the open source whylogs library. Apr 29, 2025 · Join us on Slack! 👋👋👋 Come say hi on Slack!. Optimus is an opinionated python library to easily load, process, plot and create ML models that run over pandas, Dask, cuDF, dask-cuDF, Vaex or Spark. Think of it like keeping a detailed diary of your data’s characteristics. whl; Algorithm Hash digest; SHA256: e94965eb6dbb60e2321c9e5eed3aa5ae2173338c8468f953b6229cea87a6ad89: Copy : MD5 Mar 23, 2023 · A library that provides useful extensions to Apache Spark. RayDP. io helps you find new open source packages, modules and frameworks and keep track of ones you depend upon. xls / . gz; Algorithm Hash digest; SHA256: 9962bfedf13f051340c55c19cd5138355871d2a9f06cce9065a4d78d216460e7: Copy : MD5 Apr 22, 2019 · Optimus is the missing framework to profile, clean, process and do ML in a distributed fashion using Apache Spark(PySpark). 0 onwards Data Profiling is a core step in the process of developing AI solutions. Specify the temporal metastore location. builder. head # Pearson's correlation matrix between numeric variables (pandas functionality) df. sql import SparkSession from sparkpl import DataFrameConverter # Initialize Spark spark = SparkSession. library. 1️⃣ version: 1. Type inference: automatic detection of columns' data types (Categorical, Numerical, Date, etc. transform_batch and pandas_on_spark. Description. Soda SQL is an open-source command-line tool. g. PySpark Model Conversion Tool Dec 30, 2020 · June 2024: This post was reviewed and updated to add instructions for using PyDeequ with Amazon SageMaker Notebook, SageMaker Studio, EMR, and updated the examples against a new dataset. cloud. 0: September 4th, 2017 20:58 Mar 11, 2024 · Introduction. You can also save the report to an html file. io Jun 8, 2023 · Option 1: If the spark dataframe is not to big you can try using a pandas profiling library like sweetviz, e. Jul 9, 2021 · S parkLens is an open source Spark profiling tool from Qubole which helps in tuning spark applications. 0 开始，我们很高兴地宣布，Spark 现在已经成为数据分析家族的一部分。 Profiling large datasets. PyDeequ is written to support usage of Deequ in Python. Contributions; Talk; spark-df-profiling (Q107385260) spark-df-profiling-optimus Releases 0. ydata-profiling is a leading package for data profiling, that automates and standardizes the generation of detailed reports, complete with statistics and visualizations. For each column the following statistics - if relevant for the column type - are presented in an interactive HTML report: All operations are done efficiently Spark dataframes support - Spark Dataframes profiling is available from ydata-profiling version 4. 7 or >= 3. Source Distribution Jul 5, 2022 · Hashes for spark_jdbc_profiler-1. It is the first step — and without a doubt, the most important Learn more about spark-df-profiling: package health score, popularity, security, maintenance, versions and more. 1-py2. Like pandas df. functions import col, when, lit from datetime import datetime, timezone from pyspark. Jul 18, 2020 · Optimus is the missing framework for cleaning and pre-processing data in a distributed fashion with pyspark. gz; Algorithm Hash digest; SHA256: 9962bfedf13f051340c55c19cd5138355871d2a9f06cce9065a4d78d216460e7: Copy : MD5 Dec 13, 2024 · from pyspark. Jan 1, 2013 · Hashes for spark_df_profiling-1. data_profiling_based_quality_rules (config_df, list_of_columns_to_be_ignored) 3 Nov 26, 2024 · %pip install ydata-profiling --q from pyspark. set_scan_definition_name ("Databricks Notebook") scan. I am getting the following error: 'module' object has no attribute 'view keys I am running python 2. Lux is a Python library that facilitate fast and easy data exploration by automating the visualization and data analysis process. Super Easy! df. html") # Will generate the report into a html file Generates profile reports from an Apache Spark DataFrame. corr # get the phi_k correlation matrix between all variables df. Dec 13, 2023 · The function uses our function `dqr = dq_report(df)` to generate a data quality report for each dataframe and compares the results using the column names from the report. Define checks: Use code or configuration files to specify additional validations. items Jun 3, 2019 · Steps to read . io helps you find new open Jun 9, 2022 · Hashes for soda_core_spark_df-3. gz; Algorithm Hash digest; SHA256: 0dd383dccc83c2cc5ba75a6a9b70a233e02d3eb1fdccbf920d5f438b628119e7: Copy : MD5 Note: Don't forget to load the HCC dataset. Constraints are rules or conditions that specify the expected characteristics of the data in a dataset. csv ("employe. onData returns a ColumnProfilerRunBuilder result = ColumnProfilerRunner(spark) \ . pyspark_eda is a Python library for performing exploratory data analysis (EDA) using PySpark. 12 1. 4. File details. 5 Jun 4, 2020 · A pandas-based library to visualize and compare datasets. 1 Oct 27, 2022 · File details. 1 or newer have two ways to generate data profiles in the Notebook: via the cell output UI and via the dbutils library. Installation. Source Distribution Apr 1, 2025 · PyDeequ. whylogs Overview What is whylogs . File metadata Feb 17, 2023 · Data profiling is known to be a core step in the process of building quality data flows that impact business in a positive manner. "PyPI", "Python Package Index", Jun 21, 2024 · pyspark_eda. It provides a whole report on the compute resources, wastage, data skewness, number of tasks and helps in identifying opportunities in performance optimization. The report consists of the following: DataFrame overview, Apr 14, 2025 · Gone are the days of black-box dataframes in otherwise type-safe code! Pandantic builds off the Pydantic API to enable validation and filtering of the usual dataframe types (i. Mar 14, 2025 · Spark Rapids ML (Python) This PySpark-compatible API leverages the RAPIDS cuML python API to provide GPU-accelerated implementations of many common ML algorithms. Logging data: The core of whylogs is its ability to log data. You can read the excel files located in Azure blob storage to a pyspark dataframe with the help of a library called spark-excel. It helps to understand the… May 1, 2023 · Optimus. Nov 18, 2024 · Spark Time Series Utility Package. analyze(source=(data. In a virtualenv (see these instructions if you need to create one):. Python library which makes it possible to dynamically mask/anonymize data using JSON string or python dict rules in a PySpark environment. Feast (Feature Store) is an open source feature store for machine learning. The significance of the package lies in how it Как исправить ModuleNotFoundError: No module named spark-df-profiling ошибку в python? Вы получаете эту ошибку, так как пытаетесь импортировать модуль spark-df-profiling, но он не был установлен в Вашем python окружении. UDFs enable users to… ydata-profiling的主要目标是提供一行代码的探索性数据分析（EDA）体验，以高效和一致的方式实现。就像pandas中的df. ydata-profiling 4. Dec 7, 2021 · Data teams working on a cluster running DBR 9. gz; Algorithm Hash digest; SHA256: db7ad092b66dea00974b51fea6580ba2be3952c350a1acf7b25322800e052041: Copy : MD5 May 7, 2025 · scan. csv. 6. Dec 13, 2024 · from pyspark. This library is intended for performing unit testing with PySpark on small DataFrames with functions similar to Pandas' testing module. Simple unit testing library for PySpark. 1. Jan 31, 2023 · 🎊 New year, new face, more functionalities! Thank you for using and following pandas-profiling developments. apply_batch; Type Support in Pandas API on Spark. For small datasets, the data can be loaded into memory and easily accessed with Python and pandas dataframes. Oct 26, 2023 · ydata-profiling primary goal is to provide a one-line Exploratory Data Analysis (EDA) experience in a consistent and fast solution. 1 and 0. Data Comparator is a pandas-based data profiling tool for quick and modular profiling of two datasets. Let’s begin by understanding the important characteristics of whylogs. which rows to add, delete or change to get from one dataset to the other. toPandas(), "EDA Report")) my_report. pandas-on-Spark DataFrame and Pandas DataFrame; Type Hinting with Names; Type You signed in with another tab or window. 2 (2016-07-26) / Apache-2. import sweetviz as sv my_report = sv. Soda Spark is an extension of Soda SQL that allows you to run Soda SQL functionality programmatically on a Spark data frame . Details for the file streamlit_pandas_profiling-0. ydata-profiling now supports Spark Dataframes profiling. AWS Glue Data Quality is built on Deequ […] Feb 20, 2025 · pip install duckdb-spark ## Usage ``` bash from pyspark. Spark is a unified analytics engine for large-scale data processing. Jun 21, 2023 · Like pandas df. Mar 31, 2023 · PySpark Assert. gz; Algorithm Hash digest; SHA256: 5d1c3b344823ef7bceb58688d9702c249fcc064f776b477a0aca05c01dd90d71: Copy : MD5 Nov 7, 2024 · Hashes for pyspark_pdf-0. [ ] Sep 1, 2023 · Installation of Pandas Profiling. Overview. pip安装ydata-profiling 复制PIP 这些详情尚未通过PyPI验证 Like pandas df. 项目描述; 发布历史; 下载文件; 已验证详细信息 这些详细信息已由 Dec 9, 2024 · PyPMML-Spark. SparkKG-ML is specifically built to bridge the gap between the semantic web data model and the powerful distributed computing capabilities of Apache Spark. Jan 17, 2025 · Data profiling: Automatically generate quality rule candidates with statistics. Pyspark-flame hooks into Pyspark's existing profiling capabilities to provide a low-overhead stack-sampling profiler, that outputs performance data in a format compatible with Brendan Gregg's FlameGraph Visualizer. profile_report() for quick data analysis. pip install spark-df-profiling-optimus Usage. Yet, we have a new exciting feature - we are now thrilled to announce that Spark is now part of the Data Profiling family from version 4. gz; Algorithm Hash digest; SHA256: db7ad092b66dea00974b51fea6580ba2be3952c350a1acf7b25322800e052041: Copy : MD5 Feb 14, 2025 · sparkpolars. gz; Algorithm Hash digest; SHA256: b1e7800c12099cc70de7131c959b016179dcf64f843d93d390d147ddfd3cdd5e: Copy : MD5 Oct 8, 2024 · Hashes for spark_sdk-0. cszvaumjfhuxslinqhtloyrwmhzfuqshrimmmebblylbduwmr