\documentclass{article}
%\usepackage{stix2}
\pdfoutput=1
\usepackage[utf8]{inputenc}
\usepackage[a4paper, margin=1in]{geometry}
\usepackage[active]{srcltx}
\usepackage{babel}
%\usepackage{stix}
\usepackage{verbatim}
\usepackage{mathtools}
\usepackage{bm}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{microtype,comment}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
%\usepackage{subfigure}
\usepackage{booktabs} 
\usepackage{verbatim}
\usepackage{mathtools}
\usepackage{bbm}
\usepackage{amsmath}
\usepackage{amssymb} 
\usepackage{amsthm}
\usepackage{babel}
\usepackage{amsthm}
\usepackage{dsfont}
\usepackage{array}
\usepackage{mathrsfs}
\usepackage{color}
\usepackage{natbib}
\usepackage{enumerate}
\usepackage{comment}
\usepackage{url}[sf]
\usepackage{needspace}
\usepackage{enumitem}
\usepackage{etoolbox}
\usepackage[2col]{optional} % 2col/arxiv
\DeclarePairedDelimiterX\set[1]\lbrace\rbrace{\def\given{\;\delimsize\vert\;}#1}

\newlist{pfparts}{description}{1}
\setlist[pfparts,1]{%
  font=\normalfont\textsf,
  itemindent=2pt,
  wide,
  itemsep=0pt,topsep=2pt,
  labelsep=0.75ex
}


\usepackage{tikz}
\usepackage{tikzscale}
\usepackage{pgfplots}
\usepackage{lineno}


%\usepackage{style}

\pgfplotsset{compat=1.17}
\usetikzlibrary{calc}
\usetikzlibrary{positioning}
\usetikzlibrary{intersections}
\usetikzlibrary{backgrounds}

 \bibpunct[, ]{(}{)}{,}{a}{}{,}%
 \def\bibfont{\small}%
 \def\bibsep{\smallskipamount}%
 \def\bibhang{24pt}%
 \def\newblock{\ }%
 \def\BIBand{and}%


% \def\lowint{\mkern3mu\underline{\vphantom{\intop}\mkern40mu}\mkern-20mu\int}
% % %\def\lowiint{\mkern3mu\underline{\vphantom{\intop} \vphantom{\intop} \mkern7mu}\mkern-10mu\iint}

%  \def\upint{\mathchoice%
%     {\mkern3mu\overline{\vphantom{\intop}\mkern40mu}\mkern-20mu}%
%     {\mkern7mu\overline{\vphantom{\intop}\mkern40mu}\mkern-14mu}%
%     {\mkern7mu\overline{\vphantom{\intop}\mkern40mu}\mkern-14mu}%
%     {\mkern7mu\overline{\vphantom{\intop}\mkern40mu}\mkern-14mu}%
%   \int}

% \newcommand{\lowint}[1]{
%   \underline{\int_{#1}}
% }
% \newcommand{\lowint}[2][\text{}]{
%   \underline{\int_{#1}^{#2}}
% }

% \newcommand{\upint}[1]{
%   \overline{\int_{#1}}
% }
% \newcommand{\upint}[2]{
%   \overline{\int_{#1}^{#2}}
% }
\newtheorem{example}{Example}

% \theoremstyle{plain} 
% \newtheorem{lemma}{\textbf{Lemma}} 
% \newtheorem{proposition}{\textbf{Proposition}}
% \newtheorem{theorem}{\textbf{Theorem}}\setcounter{theorem}{0}
% \newtheorem{corollary}{\textbf{Corollary}} 
% \newtheorem{assumption}{\textbf{Assumption}}
% \newtheorem{example}{\textbf{Example}} 
% \newtheorem{definition}{\textbf{Definition}}
% \newtheorem{fact}{\textbf{Fact}}
% \newtheorem{property}{Property}
% \theoremstyle{definition}
% \theoremstyle{remark}
% \newtheorem{condition}{Condition}
% \newtheorem{claim}{Claim}
% \newtheorem*{claim*}{Claim}
% \newtheorem*{lemma*}{Lemma}
% \newtheorem*{theorem*}{Theorem}
% \newtheorem{case}{Case}
% %\theorempostwork{\setcounter{case}{0}}
% \newtheorem{conjecture}{Conjecture}
% \newtheorem{remark}{\textbf{Remark}}
%  \newcommand{\vect}{\textbf} 

\usepackage{braket}
\newcommand{\vect}{\textbf} 

%\theorempostwork{\setcounter{case}{0}}
\DeclarePairedDelimiter\ceil{\lceil}{\rceil}
\DeclarePairedDelimiter\floor{\lfloor}{\rfloor}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\defeq}{:=}
\newcommand{\rhovaname}{\rho_{V}}
\newcommand{\rhovbname}{\rho_{VB}}
\newcommand{\rhova}[1]{\rhovaname\left( #1 \right)}
\newcommand{\loRiemannint}[2]{
  \underline{\int_{#1}^{#2}}
}
\newcommand{\rhovb}[1]{\rhovaname\left( #1 \right)}
\newcommand{\Aball}[2]{\mathcal{B}_{#1,#2}}
\newcommand{\Aballsize}[2]{\left| \Aball{#1}{#2} \right|}
\newcommand{\ballsize}[1]{\left| \mathcal{B}_{#1} \right|}

\definecolor{mor}{rgb}{0.8, 0.5, 0.0}
%\newcommand{\zhihao}[1]{ {\color{mor} [ Zhihao: {#1} ]  } }
\newcommand{\sss}[1]{ {\color{orange} [ Sahasrajit: {#1} ]  } }

%\newcommand{\ak}[1]{ {\color{mor} [ Aleksandra: {#1} ]  } }

 \makeatletter
\def\namedlabel#1#2{\begingroup
    #2%
    \def\@currentlabel{#2}%
    \phantomsection\label{#1}\endgroup
}
\makeatother

\makeatletter
\let\original@footnote\footnote
\newcommand{\align@footnote}[1]{%
  \ifmeasuring@
    \chardef\@tempfn=\value{footnote}%
    \footnotemark
    \setcounter{footnote}{\@tempfn}%
  \else
    \iffirstchoice@
      \original@footnote{#1}%
    \fi
  \fi}
\pretocmd{\start@align}{\let\footnote\align@footnote}{}{}
\makeatother

\title{\textbf{Homework on machine learning} }
\begin{document}
\maketitle

%In this problem we study a problem of representation robustness in participatory budgeting. 

% \textbf{Homework deadline: 5 pm on 2 May 2024(Thursday)}


\noindent
Electronic submission to Gradescope due on {\bf 5:00pm Wednesday 5/15}.
% \section{Solving optimization problems in python}

% Familiarise yourself with solving optimization problems in python. You may use some common softwares like \cite[CVXPY]{diamond2016cvxpy}, \cite[OR-Tools]{or-tools}. You may visit \url{cvxpy.org} for some examples on how to use cvxpy for the formulation of convex optimization problems. To gain more familiarity, formulate and solve the following problem on an LP solver. A linear program is formulated and solved in the colab notebook \\
% \url{https://colab.research.google.com/drive/1dAQF02yok3Y7SUUtoMq69_Hn7DYp8Xxf?usp=sharing}. 

The required python scripts are formulated and described in the colab notebook \\\url{https://colab.research.google.com/drive/1XnRa3XcHKAz4hC9jFKgUpPzjyK_M1FHn?usp=sharing} and \\\url{https://colab.research.google.com/drive/1vFs6ECjiSBGdY3PCG6G9yO_RQVzfyup_?usp=sharing}..

Please do not directly edit the colab notebook. Make a copy of the notebook and then solve the problems on it. While submitting please attach the relevant python scripts and/or excel files 
to enable us to give partial credits. Please send the required excel files to us (on the staff mailing list \url{msande214-spr2324-staff@lists.stanford.edu}) on email and paste the required urls for the colab files on the submitted pdf.

%An example how to install cvxpy in python in described in \url{cvxpy.org} 
\section*{Problem 1: Linear Regression}

We use the Boston housing dataset from \cite{scikit-learn}, which is a well-known dataset for regression tasks. It contains information about various houses in Boston, including features like the number of rooms, property tax rate, pupil-teacher ratio, etc., with the target variable being the median value of owner-occupied homes in $1000$s. In this homework we expect you to train a regression model to obtain the weight vector $\vect{w}$ (denoting the coefficients corresponding to the features) and bias $b$ (constant term) on python using optimization solvers. Train the model with and without regularization and report the difference. The files to download the dataset and test-train split are given in the colab file.

\textbf{Note}: This Boston housing data has an ethical issue \url{https://scikit-learn.org/1.0/modules/generated/sklearn.datasets.load_boston.html} and will be discussed in the lecture.

\section*{Problem 2: Pattern classification}


Consider the following data from the attached file ``data.csv'' with four columns corresponding to the label, x-coordinate , the y-coordinate and z-coordinate. The file consists of 100 rows with each row corresponding to one point. Observe that we just have two labels (0 and 1) in the entire csv file.

Solve a linear program to check whether this data points corresponding to the first label and those corresponding to the second label can be separated from each other by a hyperplane. Also try plotting the data on python (using plt.scatter) or any other visualization software.

\textbf{Note}: Since, each data point has 3 dimensions, it is fine to give three plots with two dimensions plotted each time. An example is given in the colab file.


\section*{Problem 3: Iris classification using logistic regression}


Consider the iris data from \cite{scikit-learn} stored in file ``iris-biodata-hw2.xlsx'' which consists of two different types (versicolor and setosa) of irises' with four features for every data point. Let us denote every data point by $\vect{u}^i \in \mathbb{R}^4$ for every $i \in [N]$. We now consider a model with parameters $\vect{P} \in \mathbb{R}^4$ (corresponding to the coefficients for every feature) and constant term $\alpha \in \mathbb{R}$. 


Recall that for pattern classification in class, we defined the penalty to be $\max(0,-\vect{P}^T\vect{u}^i +\alpha+1)$ and $\max(0,\vect{P}^T\vect{u}^i-\alpha+1)$ when the data-point $\vect{u}^i$ belongs to versicolor and setosa respectively. In this problem we shall consider a loss function based on the logistic regression model.

The penalty for the case when iris data-point $\vect{u}^i$ belongs to versicolor class is given by

\begin{equation}
    L_1(\vect{z}^i) = -\log \left(\frac{1}{1+e^{-z}}\right) \text { where } \vect{z}^i = \vect{P}^Tu^i-\alpha
\end{equation}

and when the iris-datapoint $\vect{u}^i$ belongs to setosa class is given by 

\begin{equation}
    L_2(\vect{z}^i) = -\log \left(\frac{e^{-z}}{1+e^{-z}}\right) \text { where } \vect{z}^i = \vect{P}^Tu^i-\alpha
\end{equation}

Plot the function $L_1(z)$ as a function of $z \in (-2,2)$. 

Now, train the model on excel by solving an optimization problem on excel by summing over all the penalty functions for every data point. Train the model with and without a regularisation parameter $C$ and observe the difference in the trained weight vectors.
%Now add the regularisation parameter to the loss function and observe how the weight vectors change.

Solve and report the objective and the learnt weight vectors and also report the accuracy. Also send us the excel files on email. Now argue why this loss function is ``easy'' to optimize. 

\textbf{Hint}: Can you observe from the plot that the function $L_1(z)$ can be approximated by set of tangent lines and this function is given by the upper envelope. 



\section*{Problem 4: Training a ML model using SVMs}


This problem is entire described in the following colab file \\
\url{https://colab.research.google.com/drive/1vFs6ECjiSBGdY3PCG6G9yO_RQVzfyup_?usp=sharing}. 

Solve it on colab and submit it to us by pasting the url on the uploaded pdf to gradescope.


\bibliographystyle{plainnat}
\bibliography{refs}



\end{document}