backward regression / PERMUTATION SHUFFLING / CLUSTERING

Backward elimination with Elastic Net is a feature selection technique used in building statistical models. It involves starting with all candidate variables and successively removing the least significant variables until only those that have a significant impact on the regression model remain. This process can help identify the most effective predictors and simplify the model without sacrificing predictive power.

Notice that a high condition number typically indicates the presence of multicollinearity among the predictor variables in our model. Multicollinearity occurs when one or more predictors are highly linearly dependent on others. This can make the regression coefficients unstable and very sensitive to small changes in the model or the data despite the fact that we a chose the best predictors where p-values are less than 0.05. Note of the Bypass Duct Pressure was not captured by backward regression and permutation shuffling as one of the important predictor unless plotted. Each units that passed through the optimal 'dip' has longer time cycle. Traditional feature importance methods in predictive models often look at features in isolation or in the context of a specific predictive framework, potentially missing interactions that only become apparent when data is visualized or grouped in high-dimensional space. Clustering algorithms excels in capturing these kinds of interactions by nature of its approach to grouping similar observations.

classification and regression

For our scenario where we are considering both binary classification and regression for predicting Remaining Useful Life (RUL), the decision on which model to train first depends on our specific goals and the nature of our data. 

1. Binary Classification First, Then Regression:

• Approach: First, classify whether an engine is near failure (e.g., will fail within a certain number of cycles). If classified as near failure, then use a regression model to predict the exact RUL or create survival probability model.

• When to Use: This approach is useful if we have a lot of variance in the RUL across our data set, or if it’s more critical to identify engines that are nearing failure as a priority. By filtering out the samples not near failure, we can focus our regression model on a narrower, possibly more homogenous subset of the data where the prediction of exact RUL is more relevant and possibly more accurate.

2. Regression First, Then Binary Classification:

• Approach: Start by predicting the RUL for all engines. Based on the predicted RUL, classify engines into categories such as 'near failure' or 'not near failure' based on a threshold (e.g., engines with a predicted RUL under a specific number of cycles are near failure).

• When to Use: This method might be preferable if our primary goal is to maintain a continuous monitoring system that predicts the lifespan of all engines regardless of their current state. The classification post-regression can help in decision-making processes or prioritizing maintenance schedules without missing out on the broader context of engine health.

Recommended Strategy: Given that we are working with engines where early identification of potential failures could be crucial, starting with binary classification to focus on potentially failing engines might be more practical. This way, we can apply more complex and possibly resource-intensive regression models on a targeted subset of our data where accuracy in predicting exact RUL is most critical. This approach can potentially improve the performance of our regression model by reducing noise from the engines that are far from failure.

FIRST MODEL TRAINING WITH REGRESSIVE TASK PER QUANTILE:

quantile_1 MSE: 0.3355726799710202

quantile_2 MSE: 0.337087532108279

quantile_3 MSE: 0.309688467364816

quantile_4 MSE: 0.19429625238753873 (As we approached the End of Cycle, our prediction is becoming more accurate).

Cycle Dynamics: The decreasing MSE might reflect a pattern in our data where the target variable becomes more predictable or less variable towards the end of the cycle.

Model Strengths: Our initial model is well-suited for predicting values in the upper quantiles of the target distribution. This could be valuable if we are primarily interested in understanding or forecasting the behavior of the target variable when it takes on higher values, however, our model limitation which have higher MSE values for the lower quantiles suggest that the model struggles to accurately predict the target variable when it takes on lower values which in this case, short life cycle engines as samples in our previous plots.