case statement in where clause,should not check only for specific values

i need a CASE statement in where clause:

lets say we have two tables X and Y X have 3 columns namely EmpId,Name,JoinDate,DeptID Y have 4 columns namely EmpId,JoinDate,DeptID,DeptRefCd

I've to get the results so that except for X.deptId in (1,2,3) it should check for X.JoinDate=ISNULL(Y.JoinDate,'').i.e., it have to ignore the JoinDate condition deptId= 1 ,2 ,3

I've written as below,but its showing error: CASE WHEN X.deptId NOT IN ('4','5','6') THEN X.JoinDate=ISNULL(Y.JoinDate,'') END

Please suggest.