timestamptz does store the timestamps in UTC. The difference between timestamptz and timestamp is that the former understands that time value represents a particular instant in time, whereas timestamp does not, the value of a timestamp field is essentially opaque because those values can be in any timezone and the system has no way of knowing.

    [postgres] # create table test(ts timestamp);
    CREATE TABLE
    Time: 29.807 ms

    [postgres] # insert into test values('2019-05-03 17:51:00');
    INSERT 0 1
    Time: 6.637 ms

    [postgres] # select * from test;
             ts          
    ---------------------
     2019-05-03 17:51:00
    (1 row)

    [postgres] # select ts at time zone 'utc' from test;
            timezone        
    ------------------------
     2019-05-03 20:51:00+03
    (1 row)
    
    Time: 17.506 ms

    [postgres] # select cast(ts as timestamptz)  from test;
               ts           
    ------------------------
     2019-05-03 17:51:00+03
    (1 row)
    
    Time: 0.428 ms

    [postgres] # select cast(ts as timestamptz) at time zone 'utc' from test;
          timezone       
    ---------------------
     2019-05-03 14:51:00
    (1 row)
    
    Time: 0.329 ms

    [postgres] # alter table test alter column ts type timestamptz;
    ALTER TABLE
    Time: 14.477 ms

    [postgres] # select * from test;
               ts           
    ------------------------
     2019-05-03 17:51:00+03
    (1 row)
    
    Time: 0.399 ms

    [postgres] # set timezone = 'America/Los_Angeles';
    SET
    Time: 9.281 ms
    
    [postgres] # select * from test;
               ts           
    ------------------------
     2019-05-03 07:51:00-07
    (1 row)
    
    Time: 0.364 ms

All that said, I think it's unfortunate that timestamptz stores values as UTC and I think it would be better for it to store the actual timezone for reasons others have already mentioned in this thread.