अफ़सोस

तजुर्बों में खुद को यूँ घोला तो था, खुद से अमूमन ये बोला तो था
खूब तन्हाई में खालीपन है, खला है, फिर भी बदतर है इश्क़, दूर रहना भला है

था भरोसा हमें हो गए हैं सयाने, नाजाने फिर दिल आज कैसे जगा है
है डरता बहुत फिर भी अपनी चलाता, अफ़सोसन मुझे इश्क़ होने लगा है 

खोज

किसी खोए हुए किस्से के खोए हुए बंज़ारे सा

खोया हुआ हूँ ख़्वाबीदा ख़यालों में इस तरह

जैसे सालों पुराना ऐब कोई खुदबख़ुद खो जाता है

Commotion

On Friday evenings after work,

while walking to the Subway station,

brushing aside pangs of anxiousness,

I stand outside the Rockefeller,

and look at the people

looking at the famous X-Mas tree.

It’s a swarm of selfie sticks.

At every step,

I hesitate.

I wouldn’t want to ruin

anyone’s holiday picture;

“Who’s that in the background?”

It would be a minor shame.

I am not exactly a festive scene.

On 2018

I know we are not through yet, but I have a history of writing end of year posts a fair bit before the end of the year, every year. In these posts I feel impelled to hope as much as to take stock, and sometimes hope for what this year wasn't but could still be in the few days that remain.

Very early in the year, something happened that made me experience the greatest physical pain I have in my life, so far. I was hospitalized for the first time in my life, and my brother came to Princeton to take care of me. It is acute and excruciating in the moment, but in hindsight I think a physical debilitation invariably seems less daunting than a mental one. In the end all that came of it was I started appreciating my brother more than I did before.

Earlier this year, I quit my prior job, a relatively relaxed affair, living in a relatively laid-back town, and started working at a new, clearly competitive place, and with it, moved to a new, clearly fast-paced city. Mostly, I was seeking a change from some sort of stasis and stillness I had started feeling. I cannot say that I am loving the change, but I should also confess I do not quite have a solid conception of what it is that I would love.

The one thing this change has ignited, though, is to seek more change. Although there was another, probably more potent catalyst for that. Soon after I had moved, my parents came visiting me here from India. I would work from early mornings to late nights (I still do, I don't quite have an option, at least as long as I am in this job) and feel bad about not spending any time with them. In my previous job, the two or three months they would live with me used to be the highlight of my year. That 25% of the year in time, I estimate, carried 99% of the year, in terms of meaning. In any case, they did not stay the entire three months. In the second month, my mom was diagnosed with a serious illness, and my parents left for India for treatment, with my brother accompanying them, to take care of my mom. 

He spent four months in India and to be able to abandon his businesses in the US for such a long time, he had to sell them. He returned a week ago to US, my mom's recovery is now progressing well. In the months that he was in India, I felt great gratitude for him. I also questioned deeply what I was doing with my life. Now, he has returned, to what can nicely be called, not much. And again I question what I'm doing with my life. What role am I serving in my family? Why have I been so unable to help?

I still have dreams (literally, in sleep) I do not want to have, those I have had for years now. It is very disconcerting when I think about the fact that I'm still having them, although I have gotten marginally better at not entertaining that thought.

A good and a bad

For the last couple of days, I've started coming to Starbucks for getting my personal coding things done. And so far, it has been so good that I have wondered why I never tried this earlier. True, sometimes, you need the intensity of being alone in the quiet of your room and your bookshelf next to you, but a great majority of the time, you don't, and what's much more crucial is that your practice is more habit-forming. To that end, I think, a kind of quasi-alone state that you get in a cafe where you are surrounded yet by yourself, is a lot more conducive than being in a locked up room. Or so I feel, for now. We'll see.

The other thing I've realized is that good textbooks are a way better way to learn something new, or even re-learn something, than video lectures. At least for me.

I've been thinking a lot about the pursuit of money and what it does to us, lately. I've long held that the utility I, and in my opinion others too in all likelihood, derive from accumulating money tends to zero, even negative, after a certain threshold. The threshold, however, where you feel like you should stop caring about accumulating still more comes much later, and for most, never at all.

I have been thinking about the second stage lately. On the one hand I feel convinced that I do not care about any pursuit the only payoff from which is more money, I think I still have some ways to go before I can say the same about external recognition, even though from what I can tell, it is entirely frivolous, whereas having money (a reasonable amount of it, at any rate) is actually pretty darn important. And yet when I see an old classmate on LinkedIn I've always thought of myself as much smarter and hard-working than, and see that he is head of so and so fancy thing at so and so fancy company, something tells me that I cannot stop running after external recognition just yet, not until I "right that wrong". I know it is immensely ignorant and small of me, but where would I confess it if not here, to nobody and everybody?

Inflection

This is the 15th of September, 2018.
I took a decision today.
I decided to decide.

नयी नौकरी, पुरानी यादें

मई में नयी नौकरी शुरू की थी, तब से दिन भर programming ही कर रहा होता हूँ. ऐसा लगता है की पिछली company में जितनी सवा चार साल में programming की थी, उतनी यहाँ 2 महीने में कर ली है, शायद उससे भी ज़्यादा. पिछली जगह काम थोड़ा subjective था, यहाँ ज़्यादातर बस programming ही करनी रहती है. वैसे अच्छा ही है, दिमाग पर ज़्यादा ज़ोर देना पड़ता है, focused state में पूरा दिन निकल जाता है, अच्छा लगता है.

सुबह आँख खुलते ही काम पर निकल जाता हूँ, और रात घर पहुँचते फिर सोने का ही समय हो जाता है. उसके बाद भी हमेशा deadline की race में थोड़ा देर से ही code ship कर पाता हूँ. इस सब का क्या मतलब है, ये सोचना छोड़ दिया है.

मम्मी को बहुत miss करता हूँ. भगवान् उनको बहुत अच्छी सेहत दे. 

Three things that I initially thought overrated but have with the passage of time come to think very highly of

1. The Mehdi Hassan ghazal: Ranjish hi sahi

2. The Farhan Akhtar movie: Zindagi na milegi dobara

3. The Farida Khanum ghazal: Aaj jaane ki zid na karo

Inflection

This is the 30th of April 2018.
I took a decision today. 
I decided to never be dishonest with myself again. 

Definition

Wishing earnestly for an outcome that you know to be impossible, is insanity. 

Two species of aloneness

On days that I would be alone in India, in the sense of not being around family and friends, I would still run into small, individually trivial but in aggregate meaningful encounters with auto drivers, bus conductors, tea vendors, shopkeepers, ironing guy, vegetable vendors, cows, temple pujaris, street food hawkers, internet cafe owners, and random dudes on the street that I knew from one time and context to another.

On days that I am alone in the US, I am alone.

And then again, days that I'm alone in the US are much more frequent than days that I would be alone in India. And since there is this completeness to the aloneness of the US variety, I'm compelled to intensify my search for what to do with those times.

Mostly, it has helped me explore areas of study, habits of self-sufficiency, and patterns of self-development, that I probably never would have had I continued to live in India, and for which I am grateful, but once in a while, it leads you to a dark place that you either dread in the moment or an escape that you regret later.

Dread alone, and regret alone, too.

Beginnings

It was some time in the summer of 2015 that I started turning to spirituality. Looking back, it was entirely spontaneous and not in the least bit planned, or even something I aspired to at any point in my life before.

True, a couple of months prior to this, my mom had come visiting me for the summer and I accompanied her to temples quite often, and it was one such visit that prompted a great urge, but it's unlikely that the temple visits were what set off this quest. (But quest is what I can call it now; at the time it was more of a refuge for my curiosity.)

A more potent catalyst had already set it in motion, unbeknownst to me, two and a half years prior, when my relationship of more than three years with my then girlfriend had come crashing down. In the year that had followed that, there was a texture of defeated daze to the very air that I breathed. Every waking moment was filled with a kind of uncontrollable self-doubt that surrounds one the foundations of whose well-established world-view are newly shattered, as if a single overarching event swept meaningless all your life experiences, all with a calm, terrifying apathy.

Initially I tried venting out with some friends, but soon realized the futility of that exercise. It became clear to me that the depths of what I felt were unsharable; in speech or text I could at best have created a poor, elementary imitation of the reality within. When I stopped talking about it, my sorrow was complete and pure, uncluttered with imagination of what it made me look like or what its verbal expression to someone might elicit. For the first time, then, I faced my sorrow squarely, with close attention rather than haywire self-pity, and then again, and then again. Every day and every night, for years to follow. That kind of inquiry into the deepest recesses of your mental and emotional landscape does something to you, it transforms you into something new. The word that comes closest to describing it, somehow, is spiritual.

But then again, the very first steps on this path were more likely, instead, those first few steps of my life, when as a toddler I must have found myself focused with all my being on the idea that I was all by myself and it was extremely, terribly important not to fall.

Unfolding

I am aware of you in love with me.
I am aware of you in love.
I am aware of you.
I am aware.
I am. 

Sundays

I came to the library today, like every Sunday, rather aimlessly, like every Sunday, and gazed at people sitting in groups and laughing, like every Sunday, and wondered, like every Sunday, what I was doing here. But unlike every Sunday, wondered also where else might I have been this Sunday such that I wouldn't have had this question, and couldn't come up with a convincing answer, and that is why, I guess, I am, here, this Sunday, again.

Benefits of sleep

That is all very well, but, the problem with sleep is, you wake up. 

Subtle difference

He had thought about every angle of it, from every paradigm possible, in a systematic manner, for a considerable period of time, and was absolutely convinced that committing suicide was unequivocally unjustified, there were no two ways about it. But he really wouldn't mind if a truck ran over him right now making him pulp. That, to be honest, would be welcome.

Rekindlings

They met every other day, although their friendship now was like that of two people who used to know each other years ago, but whose paths had diverged so much that there was little any more to talk about, even if the fondness remained intact, or deepened, even, and now sat awkwardly between them like an empty plate devoid of food they were supposed to share, and while they did now text each other every night, there were hardly any words exchanged, just emoticons of various kinds, at once necessary and inadequate, filling in for that which cannot be replaced.

Time-travel

I sometimes remember, out of the blue, instances from decades ago, from when I was five, playing ball with another tenant's kid on the roof, who remarked that I was playing very well for someone my age, or from when I was six and wanted a cool pencil box like one of my classmate's that opened on the press of a button and had a mini-piano of sorts affixed to its lid, but I never really was able to get one for myself, or from when I was seven, and after much tenacity got my parents to buy me an umbrella that you wore like a headband rather than carry like a stick and which really caught the fancy of the girls in my class, as I knew it would. And when I'm thrown back, without asking, without demand, into these ephemeral episodes, I am five, or six, or seven again, and it is immense. And sometimes, I'm thrown back to being 23, and that is euphoria, but I can't talk about it here.

Waiting

बेक़रारी सी बेक़रारी है, वस्ल है और फ़िराक़ तारी है
जो गुज़ारी न जा सकी हमसे, हमने वो ज़िन्दगी गुज़ारी है

- Jon Elia

(1)

She will have no more

of his impatience.

If he really loved her, 

he needed to learn to wait. 

Wait how much? he asked. 

A lifetime, she said proudly.

He waited.

A lifetime.

[April 2014, Princeton]

(2)

You told me, politely,

to go away.

I did.

[June 2009, New Delhi]

छोड़ो भी

जब हम में था क़रार, मिलती थी हफ्ते एक बार
अब  जो कुछ भी न रहा, अब क्यों रोज़ आती हो?

अब तो फ़िराक को भी गए हो चली हैं मुददतें
ये कोई सलात तो नहीं जो सुभ-ओ-शाम गाती हो !

शब भर मुस्कुरा के कहती हो माफ़ किया छोड़ो भी
आँख खुलते ही मगर पल में चली जाती हो।

No non-sense man

बेमिजाज़ी

जब से तू मेरा ना रहा 

किसी से कोई गिला ना रहा 

नौकरी-पेशा, गाडी, घर-मकान 

हसरतों का ये सिलसिला न रहा 

न रहा शौक़ घर को आने का 

बाहर का भी काफिला न रहा 

झिझक

यूँ हिचकिचा के तेरी तस्वीर फिर उठाई है
जैसे लौट जाता हो कोई शख्स दरवाजे से
अब अगर गौर से देखूंगा, तो मर जाऊंगा

Random Post

I felt an impulse to come write here, even though I have to confess I have only the vaguest of ideas about what it is I'm going to write. It had been a while since I came here, so maybe I should explain my absence - even though there is no real explanation. I just had been busy studying and thought that writing blog posts wasn't the most judicious use of my time.

"Studying for what?", I'm often asked. And I never have a good answer. To be more precise, I don't have an answer that would be good enough for someone who would ask that question, because someone who assumes that one always only studies for a tangible near-term goal -- a job interview, an exam, a promotion -- would probably find it difficult to empathize with where I'm coming from.

I just study just to learn. And I learn because I couldn't live any other way. "But you could just learn from life, like people do, can't you? You don't have to be studying!" I know, I can, and I do, but there's scope for more. I learn to gather insights into things that I find hard. It is not too different from why anyone runs long distances. I'm guessing they do it because it is hard, because they don't know if they can do it, and they want to find out. It is only curiosity that drives every such action about which other people ask 'why would they do that'. Anyway, as I said, I can only answer this question in roundabout, unsatisfying ways for a person who has this question.

Recently, I've been talking and skyping with girls who are strangers to me. My parents tell me about them - they think that she and I should talk - and we do. Girls I've spoken to thus far are mostly unable to understand why someone would study, code, solve math problems just like that and more importantly why they would want to put up with such habits, and so the conversation sort of fails. On the other hand, the small minority which get this are invariably in other ways so opinionated that I imagine living with them would be a lifelong presidential debate, so the conversation fails again.

There was one particular girl who I actually met in person rather than on skype recently, about which I do have a funny story to tell, but I'll reserve that for another, a hopefully funny post.

So yeah, that's what's up as well, other than studying.

Other than that, Trump won today and I'm not saying anything about the election. Yeah, I'm not. I started having election fatigue 6 months ago, and now is time for relishing the absence of election noise around.

Oh, yeah, one more thing. I didn't get promoted. Yeah, still an associate; although some senior people did care to make aghast faces at the supposed unbelievability of it. I don't buy faces, though. Faces are deceptive. Don't base your judgments, emotions or decisions on reading faces. That's childish. Did you ask my age? I'm 30. It's OK.

Rohit is coming cross-country to NYC this weekend, although unlike all the earlier times, he will not be staying at my apartment in Princeton. He will be talking next week to youngsters at his alma-mater Columbia about start-up success, or something thereabouts. I will visit him on the weekend, maybe have food at the dhaba we used to go to every day back when I was interning in the city and sharing an apartment with him 3 years ago. Those were good times. Some sort of good times, I suppose, although when I reconsider I can recall that there are ample reasons to not categorize that time as such.

That's it I guess, given that I don't intend to unload my bucket of don't-even-get-me-started emotions on this blog.

Happy Diwali!

Wishing a happy Diwali to whoever reads this! Hope this Diwali you see the external and internal changes you want to see.

And if you only want to see changes externally, hope this Diwali you can start to want to see changes internally as well! ;)

Have a great one!

Drivel

I was reading an article yesterday which quoted Iris Murdoch writing to someone "Your mind is a country I find very agreeable". I thought that was pretty cool.

We hired a new guy in our team at work, and he brings me sweets and snacks all the time and insists I eat. And after some pretense at self-control I usually cave in.

I don't call home everyday anymore like I used to.

Summer is about to end.

I've been having some trouble with dreams these nights that I don't want to have.

Cricket last Saturday was fun, Sunday not so much.

Aimless blogging

I haven't gone to the gym in a week now. I started working out this July, and I was very pumped about it until last week, going almost every single day. In my mind I thought that the first half of the year was about intensive learning, and the second half would be about intensive fitness, and I followed through on it for almost two months. Even the odd day that I wouldn't go to the gym to exercise I would at least go for a run outside, or go to the kickboxing place and pull some punches, let some steam out.

However, it's been a week now since I stopped. I remember when I missed the first day, that was the 24th of August, and I felt entirely entitled to do that. Everyone needs rest. Then the second day I just decided, well, it's okay, I can live with a little guilt. The third day I thought something escapist - something along the lines of "it's also important to study" to rationalize my decision, and then read a book, or part of it. Clearly, I was lying to myself, the motivation was simply to laze around at home. Then this weekend, I missed not only gym but my only 2 days of cricket per week, as I had to be somewhere. What was it yesterday? Yesterday, I did some yoga, yes. It didn't exhaust me at all, so I could have also gone to the gym, but then I told myself I did yoga today, and that's good enough.

And maybe it is. I used to study so much until June, when I started working out and gradually stopped studying entirely. Now I've kind of stopped working out, but haven't restarted studying either. So what am I doing? I started watching Quantico, Priyanka Chopra's US TV show. I have to say it's nothing too mind-blowing but I'm watching it anyway, you know, just being lazy. The only silver lining to the last one week has been that I have persisted with eating healthy and haven't gone back to junk food, at least not on a regular basis.

My posts these days are neither funny nor insightful, I know. Maybe things are going well elsewhere but my writing has certainly taken a steep fall. I sometimes read my old posts, and I find them so much better than how I write now that I wonder if it was really me that wrote all that.

This coming weekend I'm traveling to Cleveland to my brother's place, and from there both of us will drive to Toronto, where some of my cousins live. I went there at this same time of the year, last year, and it was very nice. I'm looking forward to the weekend. On my way back, I might first go to DC to some of my other cousins before coming back home. That should quell my yearly need for good old cousin comforts, I think. I've already been to India earlier this year, and it wasn't that much fun after the first few days. I came to the realization that the India I go back to isn't the India I left four years ago. Actually, the moment I landed back in USA, I was as thrilled as I was about going to India when I was leaving US three weeks before.

Oh yeah, I will go to the gym today. Do some curls. My biceps just don't get any bigger, though, no matter what I do. Anyway.

Yesterday

Yesterday, I went to this temple straight from work. After a little while of just hanging about, I felt terribly enervated. I left and drove straight to a restaurant, and ordered a chicken gyro. I had turned vegetarian sometime last year, but in this moment of feeling so exhausted all of a sudden, I felt like I needed something more strengthening than something vegetarian would be. Yes, this was flawed reasoning, but I'm not trying to win a nutritionist's argument here; just stating how I felt and thought in that particular moment. So I had a gyro, which is quite a lot in itself for a dinner, and before I had finished that I ordered one more. For yourself?, the guy asked, puzzled at the second order. Yes, sir.

I finished the second one equally ravenously and went home. I don't know when I fell asleep, but it was within minutes of reaching home. Before 7 PM, to be sure. 

When I woke up, it was 7:40 AM today. 12 hours and 40 minutes. I haven't slept that long in a long time. 

Something was strange, yesterday.

Moral of the story? I don't know.

retrospection(retrospection(janmasthami))

I was reminded an hour ago, out of nowhere, just sitting on my office desk, of this: http://theunsweptcorner.blogspot.com/2006/08/retrospection.html

That was 10 years ago. I was 20, and apparently was already feeling old enough to deem it appropriate to look back on my younger self. I was ruminating wistfully over a kind of life I had already left behind, one of being crazily excited about festivals such as Janmasthami, about the little things that made it awesome and, to be sure, about my own intense participation in them. At some level, I was also thinking about my 8 year old self, how I looked, how small my hands were, how short I was and how I had look up into the sky to talk to my dad. At 20, though, I already looked largely as I do today, so that difference is perhaps not as outwardly stark when I sit down looking back now as I did then. But the nature of my Janmasthami has changed again.

At 20, Janmasthami was still being celebrated all around me. My parents observed all the little things. What I missed was my own former enthusiasm. Today, there is no Janmasthami around me, except in the abstract. The people I meet and see on a day to day basis have no clue that there is some such thing. I live in a world far removed from it. The enthusiasm for Janmasthami, though, is discernably more alive now, despite that. Despite that, or, because of that? I can't say for sure. 

I will be leaving early from work today, because google helped me locate a temple where they're having Janmasthami celebrations. I admit it won't be a Janmasthami of great personal involvement that it was back when I was 8. In all likelihood, all I'll do is drive there, hang around for a little aimless while, and come back, hopefully with some delicious prasad. And that would be gold, too.

Happy Janmasthami!

YAAR! (yet another angry rant)

I feel like a phony coming back here. I don't relate to this writing your experiences thing anymore. I continue to do it all the same because, once, in a moment of clarity and some insight, I had decided that I should do it. So I'm back, and I'm ranting.

I don't know how people deal with numerous romantic relationships - I truly find it daunting. Even one is more than enough, it seems. Yet people all around me participate in this merry go round time after time, each time with an equal enthusiasm. My own experiences only informed me that the whole thing is akin to going out of your way to shop for defeats of different shapes and sizes, and the little victories once in a while are mostly accidental and always pyrrhic.

My parents talk to me about getting married every single day. For I can't even count how many days. Yet I can't see one really good reason for why I should. "Do it for them" comes the closest to convincing me to do it, but I know that it fails on the criterion of being "a good reason". For most people, I think, they get something from this and that's why they do it. Good for them. But if I don't derive anything out of it, should I merely follow the convention?

I suppose some people feel lonely without a romantic partner, but being alone has never bothered me. In fact, a vast, vast majority of my best moments in the thirty years I've been around have occurred when I was alone. Besides, those moments were the most real. Anything in the company of other people -- no matter how much I love them or they love me -- has always had a tiny winy tinge at least of something unreal, something fake, put on. And that stuff saps energy.

I'm not asocial by any means. I have some of the best people for friends, am well liked by most people I've known. Still, to subvert the arrangement of my life to an extent where my most personal tics and antics, things that I truly believe are nobody's business but mine, are also put out for someone else's accountability for a lifetime, is an action that I don't understand why I should undertake.

Rant over.

Back injury. Excruciating pain. Daunted by I don't know what. Leaking wall. Feeling insufficient. Seeking to discover what to seek. Decently happy. Lost.Vaguely disillusioned. Heavy air. The sound of insects from afar. Crammed days. Empty evening. The books I read. Reminders and questions. The books I didn't. Whatever. All that seems significant is invisible.  A dainty collection of screams that I did not scream sits portentously atop my throat, and I'll gulp it down again. Olympics are inspiring. I hope to sleep deep tonight. Tomorrow is simple.

Happy Idiot

I've been happy, as much is normal,
and most days sleep a full 8 hours.
When I wake up, and look at folks,
it seems to me their joy is a farce.
And whenever it is they look at me
with that air of inquisitive frown
"who the hell is he talking to,
this talking-to-himself clown?",

I am talking to you.

A long, sad marriage

The husband is not a bad person. The wife is not a bad person. But it's a bad marriage. The husband and the wife have very different values. Sometimes the wife thinks that the husband, indeed, is a bad person, but regarding this she is not always sure. But even when she does not think so, she does believe the wedding was a farce, a deceit; an imposition from her adoptive guardian whose decision she now sees no reason why she has to comply with. The wife wants a divorce, and the apartment she lives in. Make no mistake about that. The husband is always disappointed in her. He wonders to himself sometimes how disloyal she is. He reminds her of the copious amounts of money he has spent on her over the years. She is tired of hearing that, and just says she feels suffocated. He tells her she would be nothing without him, that she does not appreciate how good she has it, so stop being such an ungrateful wife. Sometimes in worse language than I just used. She is sick of the condescending tone. In the past, this has led to some domestic violence on both sides, some would say the point of no return to a past life of camaraderie has been reached. Even if you don't know whether it was due to the husband's actions or the wife's, it's hard not to see that irreparable damage has been done to the trust between the two. The husband's kins call her names, and tell her she's welcome to go away, but dare not think of the apartment! The wife throws similar expletives at them. The husband has reacted to all this by monitoring her with a heavy hand - something that is only worsening the wife's sense of dismay. And the more she gets fed up, she resorts to more extreme displays of spite, every couple of years trying to stir some serious harm in the husband's otherwise largely harmonious life, which includes his work life, on which, by the way, he's been focussing a lot more the last couple of years and has actually been making great strides; and although it's not all a bed of roses elsewhere on his large estate, it's this apartment with the missus in particular that most often leaves him sleepless. The wife, on the other hand, lacks this 'other' life of relative peace and progress - for her, it's all about the divorce.

All this has been going on for many, many, many years. Ask yourself: if you were asked to opine on their marriage, would you advocate that they separate amicably, or would you still argue for sticking it out. Oh, a possibly important piece of information while you make your mind, this apartment is one she already had before she got married.

And know that the conclusion you offer, is also what you think should happen about the Kashmir problem. Let there be no incongruity.

Good day

I had the most satisfying day at work today. Since this blog is also a bit of a diary, I have to note this down. Made breakthroughs in a couple of different statistical research projects I was working on, and in addition had a great day, exposition wise, on a third macroeconomic stress model I had worked on last week. Days like this are so rare, and I'm pretty thrilled with how everything went today, especially as all three were the more creative and thoughtful of my projects. I also work on several other, more regular and tedious ones, and success on them does not quite make my day in quite the same way. Hooray!

Enough self-congratulations for now, I suppose. Must start reviewing Khan Academy Linear Algebra lessons in a few minutes now.

Half-formed things - 2

Why I stop:

1. Unrealistically ambitious goals (e.g. my plans often include 15 hours a day work sessions) which, at some point, I can't deal with anymore, and put the thing aside, to start another, temporarily more intrinsically rewarding and less tedious and freedom-restricting thing.

2. Often, I just want to escape the drudgery of the implementation part.

3. I work on multiple time-consuming projects at the same time, each of which, it is quite plausible, demand sustained and exclusive attention. By the very nature of setting my goals in this way, I set myself up for failure, because at some point many of the other things will necessarily need to be put aside for some time, to focus on one particular thing. The danger is that often I fail to return to those things later.

What are NOT the reasons I stop:

1. Perfectionism - That is not why I stop. I do understand whatever I'll do will be immensely imperfect, even before I set out to start it.

* * *

From a neuro-scientific perspective, it seems to be true that I entertain the autonomous right hemisphere a little too much at the cost of the order-following left hemisphere.

Might expand on this part, later.*

* * *

Hacks to try and fix things:

1. Set precise process-based (as opposed to goals-based) targets. For example, "I will work on the XYZ project for an hour a day", as opposed to "I will write 5 pages a day".

2. Give those targets less autonomy. For example "I will work on the XYZ project for an hour everyday, at 8 PM"

3. Don't clutter. Don't have projects XYZ, ABC and PQR all vying for attention, at 8 PM, 9 PM and 10 PM respectively. It's hard to say what the sweet spot is, but intuitively it seems that having only 2 main projects has some advantages. It avoids the possible monotony with having just one, and avoids the clutter that may come with having 3 or more.

4. Expect that things can still take a lot longer than you initially imagined, and accept that. At the outset itself, ask yourself, if this took 4 times as long as I think it will take, would I still want to do it? If the answer is yes, start.

5. Start today with something (not very ambitious one, but still, something which is not trivially small - ideally at least a 4-5 day effort) and make sure that no matter what, this one project you will most certainly finish.

* * *

So, today, I choose to complete reading "The Master and His Emissary" as the first project, not too small nor too big, and one which will also help me hopefully finish the neuro-scientific perspective part just before the hacks in this post that I said I might expand on later. I will read this book for an hour a day, every day, at 10:30 PM. Not setting any restrictions on how many days I have to finish the book. My very rough estimate is 15-16 days, but let's see.

Since I, as of today, believe that having 2 projects hits the sweet spot, I will also be completing the Linear Algebra courses on Khan Academy, one that I've been putting off for a long, long time. Again, without committing to a time for finishing it, I'll just say I'll spend 2 hours on it, every day, at 7:30 PM. My rough estimate is it should take me 8-10 days, but let's see.

Most importantly, I will NOT take up any other things while I'm still at these two.

I'll come back in a couple of weeks to report my progress on these two mini projects.

Half formed things - 1

Perhaps my biggest weakness is not seeing ideas or projects to completion. Invariably, I take a lot of interest in the problem formation, in understanding everything about the thing in great, minute detail, chalking out an algorithm for the problem, at which point, one would think, or at least I end up thinking, that all the challenging parts are taken care of, and what remains is a mechanical implementation of all the hard work done so far. And then I start on this often tedious but rather critical second part, and almost always lose interest mid-way during this part, and divert my attention to another problem, with which, too, I similarly lose interest while I'm mid-way in the implementation part, and so on.

The result is that I have a lot of incomplete things. Incomplete data science projects, incomplete essays on economics, half-read books, quit training regimens, incomplete short stories. 

Recently in a conversation of some sort I was asked what I thought my big weaknesses were, and I had replied that in the trade-off between exploitation (of acquired skills) and exploration (for learning new things), I tend to veer towards exploration more than what I think is ideal. I was asked, then, if I felt that adversely affected my precision or throughput. Now since I was asked the question in a way that gave me two options, I think it restricted me to thinking only in within the bounds of these two consequences, and after some musing, I found that it did not affect my precision as much as it did my throughput. I now think that what I was reflecting upon when I said that was a rather sugar-coated, roundabout way of saying what I'm saying in this post: that I leave stuff incomplete, except that the bounded way of thinking I had been set into prevented me from getting to it with quite this clarity. It is not the throughput, either, that suffers per se, since I'm actually making decent progress per unit time, between any two points of time, only it never feels as such, because the said progress is predominantly lopsided in the first part, and has very little in the implementation part, which actually produces tangible, touchable output.

If there is one thing I find should be my topmost priority at this stage in terms of improving in a work-ethic sense, it is to start to make sure to finish things. I will write some more on this as I get any lucky insights on how to bring about this change, but for now, a diagnosis is all I have.

In praise of satire

Loved both of these:

http://thesmartset.com/powerpoint-makes-us-stupid/

https://www.youtube.com/watch?v=8S0FDjFBj8o

Awesome stuff.

On the state of Eastern Philosophy today

One of my abiding interests is Eastern Philosophy. To be more specific, the Upanishads, which after years of study and deliberation, I'm fairly convinced had outlined in minute detail millenia ago whatever western philosophy converged to towards the mid 19th century, plus more that western philosophy will probably some day converge to, if it is able to circumvent some roadblocks it has built in to its approach. On the other hand, the problem with eastern philosophy is that little progress has been made since the ancient days, and whatever progress has been made has come after gaps of centuries. A lot of time has been wasted in centuries-long debates over inconsequential details. But even their most powerful contribution is not a recent addition: the idea that the process of learning, or of attaining truth for lack a better English translation, is clearly not an exercise to be carried out using only your intellect, a fundamental idea that western Philosophy is yet to accept, and chances are slim it ever will. Still, it must be admitted that a spirit of active debate is missing from eastern philosophy when you compare it to its western counterpart - the focus is instead on disseminating what is already out there, on eulogizing its past achievements - and is inhibiting further progress in the field. Alternately, I'm also open to the idea that progress must diminish asymptotically the closer you inch to the truth, and some of that may be at work here.

Who I'm awed by

Current favourites below. It is clearly a disservice to fit many of these people under a category, because clearly their influence and great talents are wide-ranging. In fact, the common theme in those that I'm awed by is as much breadth as depth. Salman Khan (of Khan Academy) it seems knows about everything well enough to teach it to you better than your specialist teacher. Bill Gates fluidly moves between high technology and philanthropy. Ramachandran is a brilliant and eccentric neuroscientist, but his ameteur attempts at archaeology are also very impressive. Tyler Cowen and Rajan look more like specialists in this group, but their breadth inside economics is wide ranging - unlike most economists who are limited to one or two areas. Some of it, I confess, is because of of my own inclinations in Economics. Daniel Kahneman is more of a specialist, yes, but in an area that has increased greatly our understanding of how breadth has a depth all its own. David Eagleman is an outstanding neuroscientist whose work of fiction "Sum" was equally outstanding. On a similar vein, Vikram Seth - delightful poet and mighty novelist; is also a pretty good painter; George Soros, who you might better know as a hedge fund manager, made seminal dents in the philosophy of markets. Charlie Rose is the most versatile interviewer I know - and betrays a good understanding of subjects from Literary theory to Macroeconomics to Hollywood to Brain sciences to Politics. In fact, he has been responsible for introducing me to several things I later got very interested in.

In time, I'm sure this list will change as my own interests evolve, but for now I'm pretty happy with this. There are so many others I admire a lot, but in condensing the list I know I've tended to favor those who are not just great, but whose greatness exerts itself across many disciplines, as well as into a human element I love.

Entrepreneurs:
Salman Khan
Bill Gates

Academics:
VS Ramachandran
Raghuram Rajan
Tyler Cowen
David Eagleman

Philosophy:
Iain McGilchrist
Jaggi Vasudev

Artists:
Vikram Seth
Jagjit Singh
Hugh Laurie

Eloquence:
Shashi Tharoor
Barack Obama

IOAM 2

I never quite know if I am replete or not, and it seems that most other people seem to know this. I do sense hunger. Not often - because I take my meals more or less 3 times a day so I end up not letting hunger a chance to set in - but on occasions when for whatever reasons I haven't been able to, I do feel, physically, pangs of hunger. It's not something to observe under IOAM because it is the most normal thing. I only note it because I've almost never quite known the counteracting feeling of experiencing, physically, that I am replete. I just stop out of conventional wisdom and reasonable rationing borne of external stimuli (e.g. gathered knowledge to the effect of "you're supposed to stop after eating this much.."). I wonder if it's just me, or most, or everyone.

IOAM 1

Coffee has never worked for me. Never made me more active, awake, anything. So far, it seems, probably never will. I like its taste though when mixed with milk and sugar.

Inchoate

The thoughts we entertain today enslave us tomorrow. If it is an identity, not a hypothesis, then the choice we have is not regarding whether we will be enslaved, but what we will be enslaved by.  

Calling a spade a spade

Rational discourse is standing on its last legs. After all, it is such a drag in front of impressive intellectual outbursts, especially when directed against somebody well-known. Calling oneself smart is cleverly understood to be in bad taste, but hey, there's an easy fix - calling others imbeciles is fine! It obviates the need to explicitly state how you're so much more evolved. Unfortunately, this quickest road to eternal smarts is gaining widespread traction. Who needs to do the hard work of understanding complicated policy implications, when calling Rahul Gandhi a pappu suffices, never mind your own secret failures at telling stare from stair.

 Let me state my disagreement with the extent of slamming Salman Khan, the bollywood actor, is facing for comparing his physically damaging routine while shooting for Sultan, a movie where he plays a wrestler, to rape. I'm no Salman Khan fan, but I'm a fan of trying to understand things in their context. Level-headedness and humility demand that one weigh their conclusions and metaphors before spewing them out to fawning followers. And that is also why I think the comparison Salman Khan drew was ill-advised. But the same preference for carefully weighing what you say is now surprisingly amiss, also, in those who castigate him in every possible way, for what was in all likelihood more a conversational gaffe than the all-out assault on women power it is now being portrayed to be.

 Intellectual heft, today, increasingly depends on identifying the prevailing fashions of the intelligentsia and magnifying that rhetoric in passably good prose. Thinking for oneself is a minority sport, and objectivity is for losers. By his admittedly imperfect usage, did Salman Khan really mean to "trivialize" rape, or did he mean to intensify his expression of the debilitating nature of his wrestling sequences? The answer to this question is not determined by morality or sentiment, but by plain old verbal comprehension. Yes, that boring thing that we only care about when preparing for CAT. Approaching it thus, Salman Khan used a simile - and similes don't have to be perfectly logical, or perfectly concordant. In a simile one often compares A to a more widely understood thing B, often to aggrandize the effect of A. In Hindi we refer to it as Atishyokti alankar - in English, I guess, one would call that hyperbole. As far as my comprehension serves me, in comparing his training/stunts to rape, he grossly over-aggrandized, and which I'm not defending. What I intend to bring forth, in fact, is that it was much more an aggrandizement (excessive, yes, and definitely unwarranted) of his stunts, than a downplaying or trivializing of rape. Efforts to understand and portray it as primarily the latter, in my opinion, distort the reality.

 Let me take the example of the talented singer Sona Mohapatra, of whom, unlike Salman Khan, I actually am a fan, and who took to social media recently to lambaste Salman Khan, calling him stupid, nasty, dangerous and an imbecile, for trivializing rape. Except that when you call a functioning individual an imbecile, you trivialize the hardships faced by the mentally challenged. "Ah, that's silly, you're twisting her intention." Well, maybe. So how about a song of hers she posted the very same day, titled qatl-e-aam, with its recurring "aap ki aankhon mein warna aaj qatl-e-aam hai". As heinous as rape obviously is, I'm sure qatl-e-aam, or mass murder, is just as bad. So is she trivializing murder when she compares it to some arbitrary dude's eyes, insensitive to all those whose loved ones have, in the long bloody history of time, been murdered?

 "Come on, now, you're making no sense - you're just pissed with Sona Mohapatra" you might be tempted to say at this point. And I would whole-heartedly agree - that the reasoning I provide can only be considered spite with Sona Mohapatra, not a cogent advocacy of victims of mass murders. I've already made up my mind on who I want to put down, and I'm stretching things way out of context to prove my point. Alas, that is precisely the point I've been trying to make.

 In a world full of politically-correct bandwagon-jumping, calling BS is a public good. Happy to do my part. Check her song out though, it is pretty awesome: https://www.youtube.com/watch?v=vosWN0CkDP4&sns=em

On cliches

The trouble with cliches is not their content, the trouble is the fact that they're cliched, so the message passes you by without registering itself. The trouble is that you fail to really take notice of it even once because it is floated around hundreds of times. In other words, the trouble with cliches is nothing, the trouble is with you. Notice.

Time Series in R .. to be continued

library(quantmod)

SBUX <- auto.assign="F)</p" from="2012-1-1" getsymbols="" to="2012-12-31">
plot(SBUX$SBUX.Adjusted, main="SBUX Closing Price (adjusted) for 2012")

// Typically there are many things we might want to do in time series analysis
// *generate forecasts of future values (predictive distributions)
// *gain an understanding of the process itself

//Constructing Indicators
//how can a ts of past info be "transformed" into indicators useful for forecasting

//Standard ts models fit a process to the series, requiring the estimation of a few parameters

//Standard approach
//1. Plot the time series. There are a lot of features that are still most readilty identified by a human
//2. Transform original ts to a stationary ts
//(a) determine whether to apply boxcox/log transformation or not
//(b) check for trend, seasonality, sharp changes, outliers, etc then do appropriate transformation
//3. Fit a model to the stationary ts
//4. Disgnostic tests to check model is reasonable, if its not go back to step 2
//5. Generate forecasts (predictive distributions) for the model
//6. Invert the transformation made in step 2, get forecasts of original ts.

# Stationarity

//Weakly stationary ts if :
//Var(X(t))//E[X(t)]=mu for all t ---------------------------------(2)
//autocov(r,r+h)=Cov(X(r), X(r+h))= f(h) = autocov(h) --(3)

//Also (3) implies that Var(X(t)) is independent of t, that is, a constant. So (1) can be restated.

//Examples
//(1) White noise process WN(mu, SigmaSq) is stationary
//(2) Random Walk process X(t) = Summ(1,t) [WN(0,SigmaSq)] is NOT stationary
//(3) MA(1) process X(t) = WN(t) + theta*WN(t-1) is stationary
//(4) ARCH(1) process is stationary, actually ARCH(1) process is also a white noise process


acf(SBUX$SBUX.Adjusted)

Box.test(SBUX$SBUX.Adjusted,lag=10,type="Ljung-Box")

pacf(SBUX$SBUX.Adjusted)

//-----------------------------

//AR 1

//Stationarity of AR1. Stationary if |fi1| < 1

//use arima() to fit

plot(SBUX$SBUX.Adjusted)
Box.test(SBUX$SBUX.Adjusted,lag=10,type="Ljung-Box", fitdf=1) //fitdf=1 since AR(1), prevents overfitting
arima(SBUX$SBUX.Adjusted, order=c(1,0,0))

plot(diff(log(SBUX$SBUX.Adjusted))[-1])
Box.test(diff(log(SBUX$SBUX.Adjusted))[-1],lag=10,type="Ljung-Box", fitdf=1)
arima(diff(log(SBUX$SBUX.Adjusted))[-1], order=c(1,0,0))

//AR(p)

DGS3 <- auto.assign="F)</p" getsymbols="" src="FRED">DGS3<-last p="" years="">head(DGS3diff)
DGS3diff<-diff 1="" p="">DGS3diff<-dgs3diff diff="" is.na="" p="">
pacf(DGS3diff) #AR1 not appropriate as in AR1 lags 2,3.. shouldn't matter after accounting for lag 1
#In general in PACF plot for AR(p), shouldn't see significant numbers after lag p --- Important

#acf formula is hard for AR(p) but there is function ARMAacf
#see how the ACF and PACF for an AR(2) model with fi1=0.5 and fi2=-0.2 looks:
plot(0:10, ARMAacf(ar=c(0.5,-0.2), lag.max=10), type="b", pch=16) #ACF
plot(1:10, ARMAacf(ar=c(0.5,-0.2), lag.max=10, pacf=TRUE), type="b", pch=16)  #PACF

# using auto.arima() to fit.
# 1. argument max.p sets the largest value of p to consider, max allowed is 5.
# 2. to fit a pure AR(p) model, either use arima() or in auto.arima() specify max.q=0
# 3. ic="aic" or "bic", or "aicc" which is corrected AIC - used for model selction by auto.arima()
# 4. input data to this fn should be vector, not xts. Apply as.numeric()
# 5. stepwise=FALSE to be specified if you want exhaustive search
# 6. seasonal=FALSE if don't want to consider models with seasonal components (default)
# 7. argument max.q sets the largest value of q to consider, max allowed is 5.
# 8. d can be specified explictly (e.g. d=2) else it will determine itself (not preferred).


# Consider daily trading volume of MSFT for 2012, find first difference of series.
# what order AR is chosen with aicc?

# Answer

library(forecast)
MSFT <- auto.assign="F)</p" from="2012-1-1" getsymbols="" to="2012-12-31">MSFTVolumeDiff <- diff="" olume="" p="">auto.arima(as.numeric(MSFTVolumeDiff), max.p=6, ic="aicc")

//It fit a model with p=1 and q=1. In addition, mean parameter was found to be unnecessary.

//Stationarity of AR(p)
//Recall that root of a fn f(x) is value of x that makes f(x) equal 0.
//It can be shown that AR(p) is stationary if EVERY root of f(x) = 1 - Summ:n(1,p)[FI(n)*x^n]
//has absolute value greater than 1.
//To get the roots of f(x) shown above we can use in R, polyroot(c(1, -FI(1), -FI(2),...))

# for example, lets say an AR(2) with FI(1) -1.2074 and FI(2) -0.2237 is fit. Is it stationary?
polyroot(c(1,-1.2074,0.2237))
# both are greater than 1, but the first one is pretty close.
# Ruppert says, "since the roots are estimated with error, reason to suspect nonstationarity"

# Unit Root Tests
# These are hypothesis tests for stationarity
# the simplest one being Dickey-Fuller test - but useful only for AR(1)
# Augmented Dickey Fuller (ADF) is more commonly used: adf.test()

library(tseries)
adf.test(as.numeric(DGS3diff), alternative="s") #"s" for stationarity, "e" for explosive
#p-value of 0.01 suggests we can reject the null in favor of alternative, that is "stationary"

# Lets see what auto.arima fits on DGS3diff
auto.arima(as.numeric(DGS3diff), max.p=6, ic="aicc")
# It fit ARIMA(2,0,2) with some parameters. Can't use polyroot on it since it has MA component.

// MA, ARMA, ARIMA

# MA(q) process - Stationary, no correlation beyond lag q in ACF plot.

# Lets simulate MA(2) process data of length 250 with theta1=0.3 and theta2=-0.2 and sigmaSq=1
simdat <- arima.sim="" model="list(ma=c(0.3,-0.2))," n="250," sd="sqrt(1))</p">
# See if the simulated data is stationary by using ADF test
adf.test(as.numeric(simdat),alternative="s") #small p value suggests stationary data

# See if any correlation beyond lag 2 in ACF plot
acf(simdat) # no correlation beyond lag 2, as expected, since data was simulated from MA(2) process

# Let's fit a MA(2) model to this data (as if we didn't know it was simulated data from specified MA(2))
holdfit <- arima="" order="c(0,0,2))</p" simdat="">holdfit # returns an MA(2) with theta1 0.24, theta2 -0.16 and sigmaSq 0.88

# But the theta1 estimate has a std error - lets see the confidence interval for it
holdfit$coef[1] - 1.96*sqrt(holdfit$var.coef[1,1])
holdfit$coef[1] + 1.96*sqrt(holdfit$var.coef[1,1]) # 95% C.I for theta1 is (0.11,0.35)

#Ljung-Box Test (H1: "at least some correl at some lag") applied to residuals of the fit
Box.test(holdfit$residuals, fitdf=2, lag=20, type="Ljung-Box") #p not small enough to reject null-> no correl
acf(holdfit$residuals) # acf confirms box test result
qqnorm(holdfit$residuals) # normal probability plot.. need to make this work.
qqline(holdfit$residuals)

# Comparing acf from simulated data to acf from true process
acf(simdat, cex.axis=1.3,cex.lab=1.3,main="") # Sample ACF of simulated data
trueacf=ARMAacf(ma=c(0.3,-0.2), lag.max=30) # true ACF
points(0:30,trueacf, pch=16, col=2)
fitacf=ARMAacf(ma=holdfit$coef[1:2],lag.max=30) #ACF of the fitted model
points(0:30,fitacf, col=4)
legend(23,1,legend=c("True ACF","Fitted model ACF"), pch=c(16,1),col=c(2,4), xjust=1, cex=1.3)

# fitting ARIMA

DTWEXM <- auto.assign="F)" dollar="" getsymbols="" index="" p="" src="FRED" trade-weighted="">DTWEXM <- dtwexm="" p="">plot(DTWEXM) # clearly not stationary
plot(diff(DTWEXM,1,1)) #diff(vec,n,d) is lag n difference (X(t)-X(t-n)) applied d times #now looks stationary
plot(diff(DTWEXM,1,2)) #2nd (lag-1) differences .. this too looks stationary.

# We want smallest d that's good enough
adf.test(as.numeric(diff(DTWEXM,1,1)[!is.na(diff(DTWEXM,1,1))]), alternative="s")
adf.test(as.numeric(diff(DTWEXM,1,2)[!is.na(diff(DTWEXM,1,2))]), alternative="s")
# Looks like 1st differencing was sufficient, so d=1

# REVISION: using auto.arima() to fit.
# 1. argument max.p sets the largest value of p to consider, max allowed is 5.
# 2. to fit a pure AR(p) model, either use arima() or in auto.arima() specify max.q=0
# 3. ic="aic" or "bic", or "aicc" which is corrected AIC - used for model selction by auto.arima()
# 4. input data to this fn should be vector, not xts. Apply as.numeric()
# 5. stepwise=FALSE to be specified if you want exhaustive search
# 6. seasonal=FALSE if don't want to consider models with seasonal components (default)
# 7. argument max.q sets the largest value of q to consider, max allowed is 5.
# 8. d can be specified explictly (e.g. d=2) else it will determine itself (not preferred).
# 9. approx=F tso that true maximum likelihood estimates are found

fitted_model<-auto .arima="" a="" arima="" as.numeric="" d="1," fit="" ic="aic" it.="" it="" p="" see="" that="" to="" we="">
#---------------------------------------------------------------------------------------

# Forecasting

# REVISION: Standard approach
//1. Plot the time series. There are a lot of features that are still most readilty identified by a human
//2. Transform original ts to a stationary ts
//(a) determine whether to apply boxcox/log transformation or not
//(b) check for trend, seasonality, sharp changes, outliers, etc then do appropriate transformation
//3. Fit a model to the stationary ts
//4. Disgnostic tests to check model is reasonable, if its not go back to step 2
//5. Generate forecasts (predictive distributions) for the model
//6. Invert the transformation made in step 2, get forecasts of original ts.

k=3 # number of future predictions
# Rem that holdfit was the model fitted to MA(2) simulated values
next_k_preds <-predict holdfit="" n.ahead="k)</p">
# Lets do another example

AAA <- auto.assign="F," getsymbols="" p="" src="FRED">AAA <- aaa="" p="">
plot(AAA) # Clearly not stationary, there is a trend
plot(diff(AAA)) # looks like trend has been removed, could be stationary
adf.test(as.numeric(diff(AAA)[-1]),alt="s") # small p value suggests stationary
holdfit <- a="" approx="F)" arima="" as.numeric="" auto.arima="" d="1," fits="" it="" p="" process="">
#diagnostic tests
Box.test(holdfit$residuals, fitdf=1, lag=10, type="Ljung-Box") #high p-value, fail to reject null: no correl
acf(holdfit$residuals)# suggests no serial correlation among residuals
pacf(holdfit$residuals)# suggests no serial conditional correlation among residuals
plot(holdfit$residuals) # looks good

#generate forecasts
holdpreds <- 6="" also="" care="" holdfit="" inverting="" n.ahead="6)" note="" of="" p="" predict="" r="" step="" takes="" that="" the="" transformation="">holdpreds

#--------------------------------------------------------------------------------------------

# Seasonality

HOUSTNSA <- auto.assign="F)</p" getsymbols="" src="FRED">HOUSTNSA <- houstnsa="" p="">plot(HOUSTNSA, ylab="Housing Starts (thousands of units)")

# We notice that there are trends not stationary
# We also notice seasonality
acf(HOUSTNSA) # confirms seasonality, peaks every 12 lags

plot(diff(HOUSTNSA)[-1]) #this removes the trend, but seasonality remains acf(diff(HOUSTNSA)[-1])
plot(diff(HOUSTNSA, 12,1)) #this removes seasonality but trends remain. (seasonally adjusted)
plot(diff(diff(HOUSTNSA, 12,1))) # this removes both and should be good. ------------(A)
# we can fit ARMA to it, but in practice this turns out to not yield models with sufficient flexibility

# Multiplicative ARIMA models for accomodating seasonality (preferred)
# ARIMA((p,d,q)*(P,D,Q)s) where s is the period of seasonality (12 for a year, for example)
# Think of it as fitting ARIMA(p,d,q) on non-seasonal component and ARIMA(P,D,Q) on seasonal component of ts.
# This multiplicative ARIMA model is fitted upon the ts obtained in (A) above
#(In R you needn't do the differencings first)

# Fitting a multiplicative ARIMA model

# METHOD 1: using arima() fitting a ARIMA((1,1,1)*(1,1,0)12) to the housing data again.
holdfit1 <- arima="" order="c(1,1,1)," period="12))</p" seasonal="list(order=c(1,1,0),">
# METHOD 1: using auto.arima()

# s is typically already known, or estimated exploring the ACF, not something auto.arima() finds.
# ts object fed to auto.arima should already be in a form that specifies the period of the series, i.e. s

HOUSts<-ts an="" creates="" frequency="12)" function="" not="" object="" p="" r="" series="" the="" time="" ts="" xts=""># d, D should always be specified yourself based on inspection of series, although auto.arima can search for it
holdfit2<-auto .arima="" d="1,D=1,approx=F)</p" ts="">
# We got a better (smaller) aic than that in the totally self-specified arima() model holdfit1.

# Use it for forecasting now.
holdpreds <- holdfit2="" n.ahead="24)</p" predict="">plot(HOUSts, xlab="Year", xlim=c(0,22))
lines(holdpreds$pred,col=2)
lines(holdpreds$pred + (1.96*holdpreds$se), col=2, lty=2)
lines(holdpreds$pred - (1.96*holdpreds$se), col=2, lty=2)

#---------------------------------------------------------------------------------------------

# Heteroskedasticity

#---------------------------------------------------------------------------------------------

# ARFIMA models

#---------------------------------------------------------------------------------------------

# Revisiting Regression

# One key assumption in regression was that errors iid normal, i.e, uncorrelated with const variance
holdlm=lm(HOUSts~sqrt(HOUSts)) # just some random regression

# In a time series regression test whether they are really uncorrelated makes sense to see ACF of residuals
acf(holdlm$residuals, lag.max=10)

# Another way is the durbin-watson hypothesis test, made specifically to test serial correl in a linear model
# H0: there is no correlation. If small p-value, reject the null: there is correlation.
library(car)
durbinWatsonTest(holdlm,10)

# DW suffers from the same multiple testing problem as acf. Can use Ljung-Box on holdlm$residuals, if needed.
# In our example we can see that errors ARE correlated.

# How to deal with this problem?
# 1. might want to log transform the predictor or response variable or both in the linear model
# 2. If significant serial correlation, fit an ARIMA(p,d,q) process in place of model error, instead of WN.

# Example. Response var is 3-year tsy rate, predictor var is 1-yr tsy-rate.
OneYearTreas <- auto.assign="F)</p" getsymbols="" src="FRED">ThreeYearTreas<-getsymbols auto.assign="F)</p" src="FRED">
holdfit1 <- hreeyeartreas="" lm="" oneyeartreas="" p="">holdfit2 <- arima="" hreeyeartreas="" order="c(1,1,1)," xreg="OneYearTreas)</p">holdfit3 <- as.numeric="" auto.arima="" d="1)</p" hreeyeartreas="" xreg="as.numeric(OneYearTreas),">acf(holdfit3$residuals)

# This solves the problem of serial correlation. What about non-constant variance?
# One way is model the errors as GARCH. Unfortunately not straightforward like with ARIMA above. Follow steps:
#1. Fit a regular lm model.
#2. Fit a time series model to the residuals  from model fit above.
#3. Refit the model using weighted least squares, weights being reciprocals of conditional variances from step2
#   Conditional variance from garchFit() are found as: holdgarchfit@sigma.t
#   weights = 1/(holdfit@sigma.t^2)
# Do it yourself.

#----------------------------------------------------------------------------------------------