Incorporating the Weibull log-likelihood with Alpha and Beta as the loss function of our model, we also added the Weibull beta penalty at location=30 where the penalty starts to increase and growth of 10.0 to control how fast the penalty grows for beta values beyond the location.

RANDOM ALPHAS AND BETAS

When we use the Weibull log-likelihood function as a loss in our deep learning model, the goal is to find the best α (shape parameter) and β (scale parameter) for each iteration, aiming to maximize the log-likelihood (or equivalently, minimize the negative log-likelihood) of the observed data given the model parameters.

The Weibull distribution's log-likelihood function, given survival times T, event indicators E (where E=1 if the event of interest, like failure or death, has occurred, and E=0 otherwise), shape parameter α, and scale parameter β, is a way to measure how well our model's parameters fit the observed data. For censored observations (where E=0), the likelihood contribution is adjusted to account for the fact that the event has not been observed up to the time T.

Deep Learning Optimization Process

Forward Pass: The model predicts α and β for each observation based on its input features. This prediction is made by the neural network's architecture, which learns complex relationships between the input features and the parameters of the Weibull distribution.

Loss Calculation: The predicted α and β are used in the Weibull log-likelihood function to calculate the loss. This function compares the model's predictions with the actual observed survival times and event indicators. By using the negative log-likelihood as the loss, we essentially measuring how unlikely the observed data is under the model parameters predicted by the network.

Backward Pass (Gradient Descent): The gradient of the loss with respect to the model parameters (weights and biases in the neural network, including those affecting α and β) is computed. The model parameters are then updated in a direction that reduces the loss, using an optimization algorithm, in this case we use Adam.

Iteration: This process is repeated for many iterations (in this case 200 epochs), with the model parameters continuously being adjusted to reduce the loss. Over time, this converges towards the set of parameters that makes the observed data under the Weibull model (maximizing the log-likelihood).

After obtaining the Alphas and Betas from the trained model, we then calculate the Brier Score based on Weibull Survival Probabilities in terms of time, alpha and Beta. We got a score of Brier Score: 0.06667745.

After obtaining the Brier Score, we then proceed to merge the scaled tensor output to our original data so that we can compare the features if it is aligning with our test set. Once aligned, we compute the Integrated Brier Score for each Weibull distribution to determine the scores on each distribution in this dataset.

 weibull_pdfn Integrated Brier Score: 0.034

 weibull_sfn Integrated Brier Score: 0.16714285714285718

 weibull_cdfn Integrated Brier Score: 0.83285714285714298

 weibull_hfn Integrated Brier Score: 2.54276872764917e+47

  weibull_chfn Integrated Brier Score: 1.260373549927319e+50

The results for different Weibull functions reveal significant variability, indicating how each function's predictions align with the observed survival data.

weibull_pdf (Probability Density Function) IBS: 0.034

This is a very low IBS, suggesting that the probability density function provides good prediction accuracy for the time to event data. However, we will not use PDF for survival probability predictions directly, as PDF represents the rate of change in survival probability at each time point rather than the survival probability itself. 

weibull_sf (Survival Function) IBS: 0.16714285714285718

This score is typical for survival analysis, as the survival function (SF) directly estimates the survival probability over time. An IBS of ~0.167 indicates a decent model performance, showing that the model's survival function predictions align relatively well with the actual survival outcomes.

weibull_cdf (Cumulative Distribution Function) IBS: 0.83285714285714298

The CDF is the complement of the survival function (1 - SF), representing the probability of failure by time t. The higher IBS compared to SF suggests less predictive accuracy when directly interpreting CDF values as survival probabilities, which is expected since CDF and SF convey opposite probabilities.

weibull_hf (Hazard Function) IBS: 2.54276872764917e+47

The hazard function describes the instantaneous rate of failure at any given time, assuming the individual has survived up to that time. Such an extremely high IBS indicates that using the hazard function directly for survival probability prediction is inappropriate and leads to highly inaccurate predictions.

weibull_chf (Cumulative Hazard Function) IBS: 1.260373549927319e+50

Similar to the hazard function, the cumulative hazard function provides information on the accumulation of risk over time rather than direct survival probabilities. The astronomically high IBS further confirms that CHF is not suitable for direct survival probability predictions.

Interpretation and Recommendations

The survival function (SF) is the most appropriate Weibull function for calculating survival probabilities, as evidenced by its IBS. Survival analysis typically relies on SF for estimating the probability that an event (e.g., death for medical analysis, failure for machinery or equipment) has not occurred by a certain time.

The PDF, CDF, HF, and CHF serve different purposes in survival analysis, providing insights into the event rate, accumulated probability of event occurrence, instantaneous risk, and accumulated risk, respectively.