Python

Python jsonpickle Security Vulnerability: Understanding Arbitrary Code Execution Risks and Countermeasures ⚠️ Critical Warning: Python’s jsonpickle library contains a severe security vulnerability that allows attackers to execute arbitrary Python code. This article provides a detailed explanation of the mechanism, attack examples, and secure serialization best practices. 📋 Table of Contents Vulnerability Overview Attack Mechanism Real Attack Examples Detailed Risks Secure Countermeasures Implementation Examples Frequently Asked Questions Related Articles 🚨 Vulnerability Overview In modern web development, data serialization and deserialization are common practices. However, when these processes are not properly managed, they can introduce serious security vulnerabilities. ...

Decision trees are a versatile, interpretable machine learning algorithm that mirrors human decision-making through hierarchical conditional splits. Widely used for classification and regression tasks, they excel in scenarios requiring transparency and explainability. This article delves into the mathematical foundations, implementation strategies, and advanced considerations for practitioners. Key Characteristics Interpretability Transparent rule-based structure ideal for regulated industries (e.g., healthcare, finance). Enables feature importance analysis via split criteria. Non-Parametric Flexibility No assumptions about data distribution. Handles mixed data types (numeric, categorical) with minimal preprocessing. Multi-Purpose Utility ...

Overview In data analysis and machine learning, it is common to encounter datasets where boolean values are represented as strings like 't' (true) or 'f' (false). Converting these to Python’s True and False types makes subsequent processing and analysis much smoother. This article explains a simple function for this conversion and how to use it effectively. Sample Data Example Suppose you have the following DataFrame: import pandas as pd data = { 'Name': ['Alice', 'Bob', 'Charlie'], 'Availability': ['t', 'f', 't'] } df = pd.DataFrame(data) print(df) Output: ...

1. Invalid comparison between dtype=datetime64[ns, America/New_York] and datetime64 Error This error occurs when comparing timestamps in Yahoo Finance with different timezones. Solution To fix this error, we need to localize the timestamps to UTC before comparing them. start = np.datetime64(datetime.date.today() - datetime.timedelta(days=freq)) df.index = df.index.tz_localize(None) # Localize to UTC df = df[start < df.index] 2. Could not import the lzma module Error This error occurs when importing pandas 1.0 or later with an incomplete Python installation. Solution To fix this error, we need to install the xz module using brew and reinstall pandas. ...

Easily convert percentage values stored as strings (e.g., “85%”) into numeric values for analysis in Python! 🚀 Why Use This Function? 🤔 📊 Data Cleaning: Many datasets store percentages as strings, which are not suitable for calculations. 🧹 Preprocessing: Converting these to numbers is essential for machine learning and data analysis. Function Example 🐍 def str_to_rate(s): if pd.isnull(s) == False: return float(s.replace('%', '')) else: return s How to Use 🛠️ col = "ReplyRate" df[col] = df[col].apply(str_to_rate) 🔄 This will convert all percentage strings in the column to float numbers (e.g., “85%” → 85.0). ⚠️ Null values will remain unchanged.

What is pandas Datareader It is a library to download data online. Official Documentation Here is a list of data sources. Some require an API key. fred and stooq can obtain data without an API key. Remote Data Access Fred Economic Data Data source provided by the St. Louis Fed. Federal Reserve Economic Data Time Series Graph of Fred NASDAQ Let’s create a time series graph of NASDAQ. Also, let’s display the 200-period moving average SMA200. This can be done in just about 10 lines. ...

前処理 機械学習は前処理が8割と言われます。前処理の手法をまとめました。 欠損値の処理 データの一部数字がblankである場合、該当データを削除、または、代替値で補完します。 どのように欠損値を扱うかがポイントです。 処理としては、fillna,dropnaなどの関数で簡単に対処可能です。 欠損値の確認 df.isnull.sum() 欠損値の対応 平均値で補完 df = df.fillna(df.mean()) 中央値で補完 df = df.fillna(df.mean()) 最煩値で補完 df = df.fillna(df.mode()) 欠損データを削除 dropnaで削除する場合 df = df.dropna() 分類データの処理 アンダーサンプリング 分類を行う際、あるカテゴリのデータのみ件数が多い状況において、 そのカテゴリのデータを削除すること One-Hot-Encoding ダミー変数化 ダミー変数化とは、例えば、企業分類があった場合にそれをカテゴリ毎にゼロイチで表現することです。 分類データ 企業 Amazon Facebook Google One-Hot-Encoding Amazon Facebook Google 1 0 0 0 1 0 0 0 1 Target Encoding 各データをクラス分類してその出現頻度で置き換える方法です。 True/Falseの2値分類であれば、存在確率に置き換わります。 A Class False True False A Class 0.66 0.33 0.66 正規化・標準化 正規化 正規化は最小値を0最大値が1となるようにスケール変換すること。 ただし、外れ値を含む場合は、外れ値を最大値として、0側にデータが偏るため注意が必要。 $ X_{NORM} = \frac{X_i}{X_{max}-X_{min}} $ ...

ta-lib Ta-Libはテクニカル分析ライブラリです。 テクニカル教の信者にとってはインストール必須のツールになります。 https://ta-lib.org/ MacOS/OSX ta-libのインストール OSXにインストールする場合はbrewを使うと簡単です。 brew install ta-lib pip3 install TA-Lib Azure ta-libのインストール メモリは1G以上で実行しないとエラーになりました。 ta-libをインストールする際のみ1G以上に変更しましょう。 wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz tar -zxvf ta-lib-0.4.0-src.tar.gz cd ta-lib ./configure --prefix=/usr make sudo make install cd ../ rm -rf ta-lib-0.4.0-src.tar.gz rm -rf ta-lib pip3 install TA-Lib 参考動画

📦 Prerequisite: Python Virtual Environment sudo apt-get install python3-venv # Install Python virtual environment 🛠️ Installation Process Clone the Repository 📥 git clone https://github.com/freqtrade/freqtrade.git Enter Directory & Switch Branch 📂 cd freqtrade git checkout develop # Switch to develop branch Run Installation Script ⚙️ ./setup.sh --install ⏳ Important Notes The TA-Lib installation might take 20+ minutes on low-spec systems (tested on Azure 1CPU/1GB RAM) ☁️⏱️ Don’t interrupt the process - just let it run! ☕️ ✅ Completion You’ll see success messages when fully installed! 🎉 Check with freqtrade --version to verify installation ✔️ ...