Tips for improving multitask learning based on multiple outputs

I'm currently trying to use multi-task learning based on a multi-output model that both allows to get an output for classification and regression. However, at the moment it's staying at around 20% accuracy. I tried out multiple things including choosing multiple loss functions, weighting loss with loss_weights in keras. Further I tried to adapt my Adam optimizer to for different beta_1 and beta_2 values. Since I read that it's better to share more in case of overfitting I tried out the following architecture:

def multitask_model(optimizer='rmsprop', init='glorot_uniform'): 
    ampl_signal = Input(shape=(X_train.shape[1:]))
    phase_signal = Input(shape=(X_train_phase.shape[1:]))
    temp = Input(shape=(1,))
    moist = Input(shape=(1,))
    weight = Input(shape=(1,))    
    
    reg = Concatenate()([ampl_signal, phase_signal])
    clf = Concatenate()([ampl_signal, phase_signal])
    
    concat_signal = Concatenate()([reg,clf])


    x = Permute(dims=(1,2))(concat_signal)
        #x = BatchNormalization()(x)
    
    x = Conv1D(128, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(128, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)
    
    x = Conv1D(128, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(128, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)

    x = Conv1D(64, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(64, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)

    x = Conv1D(64, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(64, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)
    
    x = Conv1D(32, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(32, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)
    
    x = Conv1D(32, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(32, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)
    
    x = GlobalAveragePooling1D()(x)
    concatenated_features = Concatenate()([x,temp,moist, weight])#inputD
    x = Permute(dims=(1,))(concatenated_features)
activation='elu',kernel_initializer='glorot_normal')(x)
    

    classifier=Dense(1,activation='softmax')(x)
    regression=Dense(1,activation='linear')(x)
        
    mdl_classifier = Model(inputs=[ampl_signal, phase_signal, moist,temp, weight], outputs=classifier)
    mdl_regression = Model(inputs=[ampl_signal, phase_signal, moist,temp, weight], outputs=regression)
    mdl = Model(inputs=[mdl_classifier.inputs], outputs=[classifier,regression])

    
    mdl.compile(loss=['binary_crossentropy', 'logcosh'],
              optimizer = Adam(lr=0.0005,clipnorm = 1.,beta_1=0.9, beta_2=0.999, epsilon=0.00000001, amsgrad=True), 

    return mdl #mdl_classifier, mdl_regression

Do you have any other idea what else I could do?

Best regards,